Segment-Level Foundation Models for Multi-Class Urban Land Composition around Street Trees

Urban tree health depends on surrounding land conditions, yet assessing these conditions at the city scale remains labour-intensive. Supervised deep learning methods achieve high accuracy on benchmark datasets but fail to generalise across cities, sensors, and seasons due to domain shift. Foundation models such as the Segment Anything Model (SAM) and Contrastive Language–Image Pre-training (CLIP) have demonstrated zero-shot capability, yet prior applications focus on image-level scene classification or single-object segmentation. No prior work has applied CLIP at the segment level to assign multiple co-occurring land-use classes within a single aerial image, nor integrated these predictions with Geographic Information System (GIS) buffer analysis for tree-centred land composition. This study addresses this gap by developing a zero-shot pipeline combining multi-scale SAM segmentation, segment-level CLIP classification, and GIS buffer analysis. The pipeline was evaluated on 95 urban trees in Erlangen, Germany, using high-resolution aerial imagery from three sources spanning different seasons and sensors. Ground truth multi-class land-cover masks were manually annotated within concentric buffer rings for Vegetation, Building, and Road classes. Multi-scale segmentation outputs were spatially aligned and fused using PNG World (.pgw) files encoding affine transformations, consolidating fragmented detections into coherent object masks that provided CLIP with cleaner segment inputs. Aerial-specific prompts adapted CLIP templates to overhead imagery. Multi-scale fusion substantially improved segmentation coverage and reduced fragmentation compared to single-scale approaches. The pipeline achieved vegetation F1 scores of 0.68–0.73 across buffer rings, outperforming generic CLIP templates and approaching supervised performance, while critically maintaining robustness across domains without retraining, during which supervised methods degrade significantly. Building classification showed limitations within buffer zones due to visual ambiguity and class imbalance—a finding consistent with challenges reported in prior land-cover studies. Crucially, the pipeline was optimized on one imagery source and subsequently evaluated without modification on multiple alternative sensors and temporal periods, maintaining consistent performance where supervised methods require retraining. Segment-level zero-shot classification enables scalable urban tree monitoring across entire city inventories without manual annotation or retraining, providing a practical foundation-model-based tool for urban planners assessing multi-class tree-adjacent land composition.