Hidden Optimism in Concrete AI: Recipe-Aware Validation and Cross-Source Generalization of Compressive-Strength Models

Introduction: Random cross-validation is widely used to evaluate machine-learning models for concrete compressive-strength prediction. However, benchmark datasets may contain the same mixture recipe at multiple curing ages, allowing closely related observations to appear in both training and test folds. This study quantified recipe overlap and compared random, recipe-grouped, and cross-source validation to assess generalization to unseen mixtures.

Methods: The UCI Concrete Compressive Strength dataset initially contained 1,030 observations. Removing 25 exact duplicate records yielded 1,005 observations representing 427 unique seven-component mixture recipes. Ridge regression, random forest, and XGBoost were evaluated using repeated five-fold random cross-validation and GroupKFold validation, with each recipe confined to one fold. Cross-source transfer was assessed using the current BOxCrete open-data snapshot, restricted to 401 coarse-aggregate-containing observations from 72 mixtures. Prespecified metrics were mean absolute error (MAE), RMSE, and R².

Results: In random cross-validation, 73.7% of held-out observations had an identical recipe represented in training. Random-forest MAE increased from 3.48 MPa under random validation to 4.32 MPa under recipe grouping, a 24.3% increase; the mean difference was 0.84 MPa, with an empirical 95% split-to-split interval of 0.68–1.00 MPa. R² decreased from 0.907 to 0.863. XGBoost showed a similar MAE increase from 3.57 to 4.41 MPa, whereas ridge changed little. In transfer to BOxCrete, XGBoost achieved MAE 9.60 MPa and R² 0.517. Among 143 observations within UCI marginal predictor ranges, MAE remained 7.78 MPa and R² 0.467.

Conclusions: Random cross-validation can understate prediction error when the intended application involves unseen concrete mixtures. Recipe-grouped validation and cross-source testing provide a more stringent assessment of generalization. Cross-source performance also indicates that material and laboratory differences remain important deployment challenges beyond recipe overlap.