Reliable Mutagenicity Triage for Early Medicinal Chemistry: Scaffold-Aware Validation of AI Safety Models

Early medicinal chemistry requires rapid and dependable identification of potential genotoxic liabilities so that synthetic and experimental resources can be directed toward candidates with more favorable safety profiles. Machine-learning models can support this triage, but their practical value depends on whether reported performance extends to structurally novel chemotypes. Random train–test splitting may give overly optimistic estimates when related compounds occur across partitions, potentially affecting decisions about which chemical series should advance to experimental testing.

We evaluated this issue using a curated Ames dataset of 5,362 compounds. Five predictive models were assessed under stratified random splitting and a size- and class-matched Bemis–Murcko scaffold-aware partition designed to increase structural separation while preserving partition size and class proportions. Four models used ECFP4 fingerprints (logistic regression, random forest, XGBoost, and multilayer perceptron), and a simple graph convolutional network was included as a graph-based baseline. Conventional greedy scaffold splitting was additionally examined as a sensitivity analysis.

Under matched scaffold-aware evaluation, mean maximum test-to-training ECFP4 Tanimoto similarity decreased from 0.599 to 0.495, while ROC-AUC decreased by approximately 0.04–0.07 across the fingerprint-based models. XGBoost declined from 0.887 to 0.847 and the multilayer perceptron from 0.859 to 0.787. The simple GCN decreased from 0.761 to 0.741. Conventional greedy scaffold splitting produced greater chemical separation, with mean maximum similarity decreasing further to 0.402.

These findings indicate that validation design can materially influence the apparent reliability of AI-based mutagenicity screening. By controlling partition size and class balance while challenging models with structurally distinct compounds, scaffold-aware validation can provide medicinal chemistry teams with more realistic performance expectations when prioritizing compounds for experimental follow-up and reduce reliance on potentially overoptimistic model estimates.