Early Enough to Act? Event Level Evaluation of Predictive Maintenance Alarms Using the MetroPT-3 Dataset

Introduction: Predictive-maintenance studies frequently report pointwise anomaly-detection metrics that do not reveal whether alarms occur early enough for intervention or whether false alerts are operationally acceptable. This study develops an intervention-centered benchmarking framework and applies it to the publicly available MetroPT-3 compressed-air-system dataset.

Methods: Sensor observations were aggregated into one-minute statistical features. A strictly time-ordered design used February 2020 for model training, 1 March–16 April for threshold calibration, and 17 April–1 September for evaluation. Robust median absolute-deviation scoring, principal-component reconstruction error, isolation forest, and a compact multilayer autoencoder were evaluated under the same three calibration-threshold quantiles and three alarm-persistence rules. Performance measures included pointwise precision, recall, and F1 score; event-level detection of four reported failures; 24-hour early-warning recall; signed warning lead time; false-alarm events per 30 days; and cost-sensitive indices.

Results: The cost-selected isolation-forest configuration, using a 99th-percentile threshold and a six-of-ten-minute persistence rule, detected all four failure events and achieved a pointwise F1 score of 0.708 while generating 1.31 false-alarm events per 30 evaluation days. However, only one failure was identified within the preceding 24-hour warning window, with an eight-minute lead time. A more permissive isolation-forest configuration increased early-warning recall to 75% but produced 4.81 false alarms per 30 days. The multilayer autoencoder did not improve the operating frontier: with thresholds calibrated identically to the other methods, it produced no evaluation-period alarms and detected none of the four reported failures.

Conclusions: Neither strong pointwise performance nor greater model complexity necessarily provides actionable maintenance warning. Alarm timing, persistence, false-alarm burden, and intervention costs should therefore be evaluated alongside conventional accuracy metrics when benchmarking predictive-maintenance systems.