Qualification and Evaluation of Uncertainty Estimation Concepts for Automatic Change Detection

The exploitation of earth observation data plays a crucial role in analyzing and tackling some of the most pressing global issues and societal challenges as defined in the United Nations' Sustainable Development Goals, especially in fields like urban planning and monitoring for sustainable cities or climate action and natural disaster response. A recently developed multi-modal automatic change-detection pipeline allows for the hybridization of 2D and 3D satellite data in order to combine uncertainty-aware semantic segmentation maps with detected changes in altitude and to obtain fine-grained information about urbanization or affected buildings in the aftermath of natural disasters like floods, wildfires, and earthquakes. The 2D part is based on modern deep neural networks, which often suffer from under-/overconfident predictions and lack a reliable representation of uncertainty. This work compares different common uncertainty-aware deep neural networks in the context of remote sensing imagery, like approximate Bayesian neural networks, neural network ensembles, and test-time data augmentations. It includes a quantitative evaluation of different widely used uncertainty estimation approaches for modern deep neural networks in the specific use-case of leveraging earth observation data for urban monitoring and natural disaster response. The evaluation examines well-known standard metrics to assess the quality of model calibration, like the Brier Score and the Expected Calibration Error in the context of multi-temporal semantic segmentation and change detection. Although calibration metrics can give insightful information about the reliability of model predictions with the implicit consideration of associated uncertainty estimates, evaluating uncertainty itself remains a challenging task, due to a lack of ground-truth data. Different approaches for a quantitative evaluation of uncertainty estimates have been proposed in the literature, like the “Patch Accuracy vs Patch Uncertainty” metric, which is thus included in the evaluation in order to complete the quantitative comparison of different uncertainty estimation methods for EO change detection.