Dataset Bias in the Natural Sciences: A Case Study in Chemical Reaction Prediction and Synthesis Design
This work addresses dataset quality issues for researchers in chemistry and machine learning, but it is incremental as it critiques existing practices without proposing new solutions.
The paper identifies three dataset biases in chemical reaction prediction and synthesis design, such as unrealistic data splits and mislabelled data, that hinder machine learning applications in the natural sciences, and critiques their impact on experimental progress.
Datasets in the Natural Sciences are often curated with the goal of aiding scientific understanding and hence may not always be in a form that facilitates the application of machine learning. In this paper, we identify three trends within the fields of chemical reaction prediction and synthesis design that require a change in direction. First, the manner in which reaction datasets are split into reactants and reagents encourages testing models in an unrealistically generous manner. Second, we highlight the prevalence of mislabelled data, and suggest that the focus should be on outlier removal rather than data fitting only. Lastly, we discuss the problem of reagent prediction, in addition to reactant prediction, in order to solve the full synthesis design problem, highlighting the mismatch between what machine learning solves and what a lab chemist would need. Our critiques are also relevant to the burgeoning field of using machine learning to accelerate progress in experimental Natural Sciences, where datasets are often split in a biased way, are highly noisy, and contextual variables that are not evident from the data strongly influence the outcome of experiments.