Beyond Bulk Statistics: Diagnosing Amplitude, Phase, and Concentration-Dependent Errors in a Pan-European CMAQ Simulation

Bulk statistics such as mean bias and correlation are widely used to evaluate chemical transport models but can mask whether errors stem from level offsets, damped variability, or concentration-dependent behavior, a distinction critical for interpreting model reliability in policy contexts under Directive (EU) 2024/2881, which sets stricter PM2.5 and O3 limits ahead of 2030.

Using a pan-European WRF–CMAQ (v5.3, CB6) simulation for 2019, we apply a layered diagnostic approach to surface O3 and PM2.5, evaluated hourly against more than 2300 and 1700 European monitoring stations, respectively. Taylor diagrams jointly assess correlation and variance ratios; Target diagrams decompose systematic (bias) from random/phase error components normalized to observed variability; and quantile-binned error (QBE) analysis, computed across 18 equal-population percentile bins, isolates concentration-dependent behavior invisible in seasonal or annual averages. Results are further benchmarked against Emery et al. and Boylan–Russell performance criteria.

The diagnostics show that random and phase errors, not systematic bias, dominate total error for both pollutants (normalized unbiased RMSD ≈ 0.85–0.92), despite near-neutral annual O3 bias (+1.9 ppb) and a persistent negative PM2.5 bias (−5.0 µg m⁻3). QBE analysis reveals a characteristic S-shaped O3 error curve, overprediction at low, nighttime/titration-dominated concentrations and underprediction at photochemical peaks, alongside a strongly asymmetric PM2.5 error profile, with underestimation intensifying sharply toward the upper percentiles, particularly during winter over Eastern and Southern Europe, where median errors exceed 30 µg m⁻3 in the highest bins.

These results demonstrate that pattern- and distribution-based diagnostics provide information essential for assessing model suitability in extreme-event and exposure-relevant policy applications, beyond what conventional bias/correlation metrics alone can reveal.