Interpretable Multimodal Deep Learning with Rough Set Theory for Automated Neonatal Pain Assessment: An Eye-Tracking Dominant Approach

Introduction: Pain assessment in non-communicative neonates remains challenging because conventional clinical scales are subjective, show inter-rater variability, and provide intermittent observations. This study developed an interpretable multimodal deep-learning framework for continuous neonatal pain-level classification, with emphasis on eye-tracking features and Rough Set Theory (RST)-based explanation.
Methods: A prospective observational dataset included 133 infants with gestational ages of 28–42 weeks, comprising 355 synchronized multimodal recording segments and 75 h of data. Facial video, eye-tracking measures, ECG/heart-rate variability, oxygen saturation and respiratory signals, and cry audio were acquired. DeepLabV3+ generated binary masks of the facial region, separating the face from background and equipment before VGG16 facial-embedding extraction; segmentation achieved a Dice coefficient of 0.95. Pupil size, blink rate, saccade velocity, physiological variables, facial embeddings, and audio features were integrated. RST reduced 47 features to 28 variables and generated interpretable decision rules. Pain was classified into six levels (0–5), with ground-truth categories assigned by dual expert evaluation using NFCS, COMFORT, and NIPS, followed by consensus resolution. Grouped 10-fold cross-validation was performed at the infant level; all segments from an individual infant were restricted to one fold.
Results: XGBoost achieved 99.5% accuracy and an F1-score of 0.99, outperforming NGBoost (98.0% accuracy; F1-score 0.98), random forest (97.0%; 0.97), MLP (94.0%; 0.94), and SVM (92.0%; 0.92). SHAP analysis identified pupil size (mean absolute SHAP value: 0.42) and blink rate (0.35) as the two most influential predictors. Model outputs showed agreement with clinical assessments, including NFCS correlation (r = 0.92, p < 0.001), COMFORT agreement (κ = 0.78), and NIPS agreement (κ = 0.76). Mean inference time was 33.2 ms per frame, supporting real-time processing at 30 frames/s.
Conclusions: The proposed framework combines multimodal sensing, deep learning, RST feature selection, and explainable AI for accurate and interpretable neonatal pain-level classification. Eye-related features were prominent predictors and warrant further validation in independent prospective NICU cohorts.