Auto-Bench: An Automated Benchmark for Scientific Discovery in LLMs
This addresses the problem of assessing LLMs' ability to conduct scientific research for AI and ML researchers, though it is incremental as it builds on existing causal graph methods.
The authors tackled the lack of a standardized benchmark for evaluating LLMs in scientific discovery by introducing Auto-Bench, a novel benchmark based on causal graph discovery, and found that state-of-the-art LLMs like GPT-4 show a significant performance drop as problem complexity increases.
Given the remarkable performance of Large Language Models (LLMs), an important question arises: Can LLMs conduct human-like scientific research and discover new knowledge, and act as an AI scientist? Scientific discovery is an iterative process that demands efficient knowledge updating and encoding. It involves understanding the environment, identifying new hypotheses, and reasoning about actions; however, no standardized benchmark specifically designed for scientific discovery exists for LLM agents. In response to these limitations, we introduce a novel benchmark, \textit{Auto-Bench}, that encompasses necessary aspects to evaluate LLMs for scientific discovery in both natural and social sciences. Our benchmark is based on the principles of causal graph discovery. It challenges models to uncover hidden structures and make optimal decisions, which includes generating valid justifications. By engaging interactively with an oracle, the models iteratively refine their understanding of underlying interactions, the chemistry and social interactions, through strategic interventions. We evaluate state-of-the-art LLMs, including GPT-4, Gemini, Qwen, Claude, and Llama, and observe a significant performance drop as the problem complexity increases, which suggests an important gap between machine and human intelligence that future development of LLMs need to take into consideration.