When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs
This work addresses the need for more realistic evaluation in synthesis planning for drug discovery, though it is incremental as it builds on existing LLM applications.
The paper tackles the problem of evaluating retrosynthesis performance in drug discovery by proposing a new benchmarking framework that uses ChemCensor, a metric for chemical plausibility, to better align with real-world synthesis planning. It introduces the CREED dataset with millions of validated reaction records and trains a model that improves over LLM baselines under this benchmark.
Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance remains limited. Existing benchmarks and metrics typically rely on published synthetic procedures and Top-K accuracy based on single ground-truth, which does not capture the open-ended nature of real-world synthesis planning. We propose a new benchmarking framework for single-step retrosynthesis that evaluates both general-purpose and chemistry-specialized LLMs using ChemCensor, a novel metric for chemical plausibility. By emphasizing plausibility over exact match, this approach better aligns with human synthesis planning practices. We also introduce CREED, a novel dataset comprising millions of ChemCensor-validated reaction records for LLM training, and use it to train a model that improves over the LLM baselines under this benchmark.