Typical building inspection using Passive Infrared Thermography involves defect analysis. With different image processing techniques, it is possible to perform defect size estimation. However, the current approach only relies on a single thermal image to estimate the defect size, and the time when the thermal image was captured highly affects the performance. Since defects on the building’s exterior wall are in an ambient environment, the ideal scenario is to collect thermal images of the target with time which captures the whole heat absorption and dissipation period for better defect analysis. But such an approach is unfeasible in urban areas. This study proposes the adoption of the Thermal Decay Gradient Approach in which only two thermal images (captured in the heat absorption period and the heat dissipation period respectively) were used in the defect analysis. A west-facing sample wall with 10 embedded defects in different cover depths was constructed in an outdoor environment, and thermal data of the sample wall were taken once every 5 minutes in a 24-hour cycle. This study considered different combinations of the two thermal images to be processed, and then natural logarithmic and linear regression fitting was applied to the temperature values in the selected thermal images. Then the thermal decay gradient of each pixel was calculated and used to form a visualization in which the defects were highlighted, and the resulting visualization could be used for defect size estimation. This study evaluates the accuracy of the proposed method by comparing the defects’ estimated sizes with actual sizes.