Extreme drought periods have severely affected forests in Central Europe in the last few years. As a result, extended wilting, defoliation and die-out events have severely impacted several tree species. Within the framework of the ForstEO research project, we evaluated the capability of deep learning approaches to identify tree crown damage in selected areas of Bavaria, Germany. We used 20 cm aerial imagery from different years: 2019 (June and August), 2020, 2021 and 2023. We provide an analysis of the accuracy of wall-to-wall image segmentation and give insights into the feasibility of generating models that are transferable to other datasets. We evaluated the following scenarios: 1) two classes: background and damaged trees; 2) three classes: background, damaged deciduous trees and damaged coniferous trees; and 3) four classes: similar classes as in 2, but with the damaged conifer classsplit into pine and other conifers. We trained several U-Net variants for semantic segmentation using image patches and masks obtained from four different datasets, and applied the generalized models to classify a test area for the studied years. The highest mean interclass IoU for the models attained 0.82 for the 2-class, 0.73 for the 3-class and 0.69 for the 4-class cases. Among the three evaluated forest categories, detection rates varied significantly depending on the dataset. As a trend, damaged deciduous trees exhibited the highest detection values, whereas conifers demonstrated the lowest. When the best model was applied to unseen data (June 2019), IoU achieved a best mean of 0.78 for the 2-class, 0.69 for the 3-class and 0.59 for the 4-class cases. The general models showed robust performance on all datasets. However, the transferability of these models needs to be further investigated.