Procrustean Bed for AI-Driven Retrosynthesis: A Unified Framework for Reproducible Evaluation
This addresses reproducibility and benchmarking issues for researchers in AI-driven retrosynthesis, though it is incremental as it focuses on evaluation infrastructure rather than new synthesis methods.
The authors tackled the lack of standardized evaluation in computer-aided synthesis planning by introducing RetroCast, a unified framework that revealed a divergence between solvability scores and route quality, with search-based methods showing a sharp performance decay in long-range plans compared to sequence-based approaches.
Progress in computer-aided synthesis planning (CASP) is obscured by the lack of standardized evaluation infrastructure and the reliance on metrics that prioritize topological completion over chemical validity. We introduce RetroCast, a unified evaluation suite that standardizes heterogeneous model outputs into a common schema to enable statistically rigorous, apples-to-apples comparison. The framework includes a reproducible benchmarking pipeline with stratified sampling and bootstrapped confidence intervals, accompanied by SynthArena, an interactive platform for qualitative route inspection. We utilize this infrastructure to evaluate leading search-based and sequence-based algorithms on a new suite of standardized benchmarks. Our analysis reveals a divergence between "solvability" (stock-termination rate) and route quality; high solvability scores often mask chemical invalidity or fail to correlate with the reproduction of experimental ground truths. Furthermore, we identify a "complexity cliff" in which search-based methods, despite high solvability rates, exhibit a sharp performance decay in reconstructing long-range synthetic plans compared to sequence-based approaches. We release the full framework, benchmark definitions, and a standardized database of model predictions to support transparent and reproducible development in the field.