Re-evaluating Retrosynthesis Algorithms with Syntheseus
This addresses the issue of unreliable progress assessment in retrosynthesis research for chemists and ML practitioners, though it is incremental as it focuses on evaluation rather than new methods.
The authors tackled the problem of inconsistent benchmarking in automated synthesis planning by introducing syntheseus, a library with an extensive framework for consistent evaluation, and found that the ranking of state-of-the-art retrosynthesis models changes under controlled experiments.
Automated Synthesis Planning has recently re-emerged as a research area at the intersection of chemistry and machine learning. Despite the appearance of steady progress, we argue that imperfect benchmarks and inconsistent comparisons mask systematic shortcomings of existing techniques, and unnecessarily hamper progress. To remedy this, we present a synthesis planning library with an extensive benchmarking framework, called syntheseus, which promotes best practice by default, enabling consistent meaningful evaluation of single-step models and multi-step planning algorithms. We demonstrate the capabilities of syntheseus by re-evaluating several previous retrosynthesis algorithms, and find that the ranking of state-of-the-art models changes in controlled evaluation experiments. We end with guidance for future works in this area, and call the community to engage in the discussion on how to improve benchmarks for synthesis planning.