Demographic Bias in Vision–Language Pretraining for Chest X-Ray Diagnosis: An Empirical Audit and Proposed Mitigation Strategy

Deep learning models trained on chest X-ray datasets are widely used for automated screening of conditions such as pneumonia, cardiomegaly, and pleural effusion, yet growing evidence shows that these models can perform inconsistently across patient subgroups defined by sex or age. This disparity is especially concerning for vision–language pretrained models, which increasingly serve as backbones for chest radiograph classification and report generation, since biases learned during pretraining can propagate into every downstream clinical task. This work presents an empirical audit of subgroup performance gaps in a custom CLIP-style vision–language model pretrained on chest X-ray image–report pairs, in the style of existing models such as CheXzero and GLoRIA, using large-scale public datasets including CheXpert and MIMIC-CXR. The audit evaluates sex and age as protected attributes; race/ethnicity is deliberately excluded given well-documented label inconsistency and self-report unreliability in CheXpert/MIMIC-CXR demographic metadata, which would compromise the validity of subgroup analysis based on that attribute. Building on the audit findings, we propose a mitigation strategy that combines disease-consistent positive pairing with a Hilbert–Schmidt Independence Criterion (HSIC)-based regularization penalty added to the contrastive loss, designed to reduce statistical dependence between learned representations and protected attributes while preserving diagnostically relevant features for pneumonia, cardiomegaly, and pleural effusion classification. Preliminary audit results indicate measurable subgroup disparities, particularly underdiagnosis patterns in certain groups, even in high-performing baseline models; quantitative evaluation of the proposed mitigation strategy is ongoing. This study contributes a reproducible bias-auditing protocol and a concrete, mechanistically grounded mitigation approach, offering a practical step toward more equitable AI-assisted thoracic diagnosis.