Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents
This benchmark addresses the problem of evaluating AI agents' scientific discovery capabilities, particularly in physics, for researchers in AI and scientific computing, though it is incremental as it builds on existing benchmark concepts.
The paper introduces Gravity-Bench-v1, a benchmark that challenges AI agents to discover physics, specifically gravitational dynamics, within a simulated environment, and it proves difficult for baseline agents.
Modern science emerged from reasoning over repeatedly-observed planetary motions. We present Gravity-Bench-v1, an environment-based benchmark that challenges AI agents on tasks that parallel this historical development. Gravity-Bench-v1 evaluates agents on the discovery of physics concealed within a dynamic environment, using rigorous gravitational dynamics simulations. Gravity-Bench includes out-of-distribution cases, i.e. with physics that deviates from the real world, to evaluate true scientific generalization capabilities. Agents must plan to collect data within an experimental budget and must perform a dynamic form of data analysis and reasoning to solve tasks efficiently. Our benchmark admits an open-ended space of solutions. Reference solutions for each task are provided to calibrate AI performance against human expertise. Technically at an upper-undergraduate level, our benchmark proves challenging to baseline AI agents. Gravity-Bench-v1 and planned extensions should help map out AI progress towards scientific discovery capabilities.