Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab
This addresses the need for cost-effective evaluation of LLMs' scientific reasoning in biology, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating large language models' scientific capabilities in experiment design and analysis by introducing SciGym, a benchmark using simulated biological systems, and found that while more capable models performed better, all models declined significantly with increased system complexity.
Designing experiments and result interpretations are core scientific competencies, particularly in biology, where researchers perturb complex systems to uncover the underlying systems. Recent efforts to evaluate the scientific capabilities of large language models (LLMs) fail to test these competencies because wet-lab experimentation is prohibitively expensive: in expertise, time and equipment. We introduce SciGym, a first-in-class benchmark that assesses LLMs' iterative experiment design and analysis abilities in open-ended scientific discovery tasks. SciGym overcomes the challenge of wet-lab costs by running a dry lab of biological systems. These models, encoded in Systems Biology Markup Language, are efficient for generating simulated data, making them ideal testbeds for experimentation on realistically complex systems. We evaluated six frontier LLMs on 137 small systems, and released a total of 350 systems. Our evaluation shows that while more capable models demonstrated superior performance, all models' performance declined significantly as system complexity increased, suggesting substantial room for improvement in the scientific capabilities of LLM agents.