Holistic chemical evaluation reveals pitfalls in reaction prediction models
This work addresses the need for more robust evaluation in chemical reaction prediction, which is crucial for accelerating chemical discovery, though it is incremental as it builds on existing approaches.
The paper tackles the problem of limited evaluation metrics in chemical reaction prediction models by proposing a new holistic assessment scheme, which reveals important differences in model performance on stereoselectivity and out-of-distribution generalization.
The prediction of chemical reactions has gained significant interest within the machine learning community in recent years, owing to its complexity and crucial applications in chemistry. However, model evaluation for this task has been mostly limited to simple metrics like top-k accuracy, which obfuscates fine details of a model's limitations. Inspired by progress in other fields, we propose a new assessment scheme that builds on top of current approaches, steering towards a more holistic evaluation. We introduce the following key components for this goal: CHORISO, a curated dataset along with multiple tailored splits to recreate chemically relevant scenarios, and a collection of metrics that provide a holistic view of a model's advantages and limitations. Application of this method to state-of-the-art models reveals important differences on sensitive fronts, especially stereoselectivity and chemical out-of-distribution generalization. Our work paves the way towards robust prediction models that can ultimately accelerate chemical discovery.