Generalized Statistical Testing of Interchangeability in Performance Between an AI Segmentation Device and a Multi-Expert Human Panel

Introduction: AI-based medical imaging devices increasingly incorporate segmentation capabilities for lesions and anatomical structures. Conventional segmentation algorithm evaluation generally compares device output to an aggregated reference annotation, considered to be correct, using metrics such as the Dice Similarity Coefficient (DSC) or Hausdorff Distance (HD), but this type of evaluation is constrained by the absence of a true gold standard and difficulty defining clinically meaningful thresholds. A generalized paired statistical testing method addressing these limitations has been established (Hu et al., 2025, 2026); its behavior on real clinical imaging data using both overlap- and distance-based metrics simultaneously, however, remains uncharacterized and is the subject of this work.

Methods: This study applies this generalized paired interchangeability testing method to segmentation data from the Lung Image Database Consortium (LIDC) dataset to evaluate how a trained AI segmentation model compares with an expert human reader panel in terms of both overlap-based and distance-based segmentation performance.  

Results: Application of the generalized paired interchangeability testing method using DSC and HD yielded concordant conclusions. Both analyses indicated a statistically significant difference between the trained AI model-to-expert similarity (and distance) versus the expert-to-expert similarity (and distance). These findings were consistent across all comparisons between the AI model and all reader subpanels.

Conclusions: The generalized paired interchangeability testing approach was applied to the assessment of a trained AI segmentation model using real clinical imaging data and applying both DSC and HD performance metrics. Consistent findings across both overlap- and distance-based metrics strengthen the resulting inference and support the practical utility of this method for evaluating AI-based medical image segmentation devices.