Are large language models superhuman chemists?
This work addresses the need for understanding LLMs' capabilities in chemistry to improve models and mitigate harm, though it is incremental as it focuses on benchmarking existing models.
The paper tackled the problem of systematically evaluating the chemical knowledge and reasoning abilities of large language models (LLMs) by introducing ChemBench, a framework with over 2,700 question-answer pairs, and found that the best models outperformed human chemists on average.
Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here, we introduce "ChemBench," an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.