QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry
This addresses the need for domain-specific evaluation in quantitative chemistry for researchers and developers, but it is incremental as it focuses on benchmarking rather than novel methods.
The authors tackled the problem of evaluating large language models (LLMs) on rigorous quantitative chemistry calculations by introducing QCBench, a benchmark with 350 problems across 7 chemistry subfields, and found that 24 LLMs showed consistent performance degradation with increasing task complexity.
Quantitative chemistry is central to modern chemical research, yet the ability of large language models (LLMs) to perform its rigorous, step-by-step calculations remains underexplored. To fill this blank, we propose QCBench, a Quantitative Chemistry oriented benchmark comprising 350 computational chemistry problems across 7 chemistry subfields, which contains analytical chemistry, bio/organic chemistry, general chemistry, inorganic chemistry, physical chemistry, polymer chemistry and quantum chemistry. To systematically evaluate the mathematical reasoning abilities of large language models (LLMs), they are categorized into three tiers: easy, medium, and difficult. Each problem, rooted in realistic chemical scenarios, is structured to prevent heuristic shortcuts and demand explicit numerical reasoning. QCBench enables fine-grained diagnosis of computational weaknesses, reveals model-specific limitations across difficulty levels, and lays the groundwork for future improvements such as domain-adaptive fine-tuning or multi-modal integration. Evaluations on 24 LLMs demonstrate a consistent performance degradation with increasing task complexity, highlighting the current gap between language fluency and scientific computation accuracy. Code for QCBench is available at https://github.com/jiaqingxie/QCBench.