Failure Modes of a Light weight Instance-Segmentation Model for Radiographic Caries Detection: A Clinically Grounded Analysis

Introduction: Deep learning models for radiographic caries detection are often evaluated through aggregate performance metrics, which obscure the specific clinical contexts in which detection succeeds or fails. Because undetected lesions in particular anatomical and restorative settings carry distinct clinical consequences, characterizing these failure contexts is a prerequisite for safe translation. This study examines the failure modes of an instance-segmentation model in relation to the established difficulty of radiographic caries diagnosis.

Methods: A YOLOv8n-seg model, pre-trained on COCO and fine-tuned by transfer learning, was trained on a publicly available dataset (CC BY 4.0) of 552 bitewing radiographs, augmented to 1,653 images and annotated for two classes: caries (7,432 instances) and enamel (5,795 instances). Training was performed at 640 × 640 pixels for 50 epochs (batch size 16) on a single NVIDIA T4 GPU. As no held-out partition was defined, per-class mean average precision (mAP50, mAP50–95) was computed in-distribution and is interpreted as an upper-bound estimate. Error detections were examined qualitatively to identify systematic failure patterns by lesion type and location.

Results: In-distribution detection reached mAP50 0.705 for caries and 0.965 for enamel (aggregate 0.835; mAP50–95 0.601). Qualitative inspection revealed a consistent pattern: detection was most reliable for primary lesions on intact enamel and for early demineralization, whereas false negatives were consistently observed for lesions at alveolar crest level and cementoenamel junction, and in secondary caries beneath or at restoration margins, settings characterized by low or confounded radiographic contrast. These observations parallel the recognized limits of human radiographic interpretation.

Conclusions: Model failures align with regions of inherent diagnostic difficulty rather than random error, indicating that in-distribution aggregate metrics inadequately represent clinical readiness. Held-out evaluation, targeted annotation of cementoenamel junction and peri-restorative lesions, and independent clinician confirmation of suspected early lesions are required before clinical deployment.