SimuScene: Training and Benchmarking Code Generation to Simulate Physical Scenarios
This addresses the underexplored ability of LLMs to simulate physical scenarios, which is important for advancing AI in scientific and engineering domains, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the problem of training and evaluating large language models (LLMs) to simulate physical scenarios via code, finding that even the strongest model achieves only a 21.5% pass rate, but training with their data improves both physical simulation and general code generation.
Large language models (LLMs) have been extensively studied for tasks like math competitions, complex coding, and scientific reasoning, yet their ability to accurately represent and simulate physical scenarios via code remains underexplored. We propose SimuScene, the first systematic study that trains and evaluates LLMs on simulating physical scenarios across five physics domains and 52 physical concepts. We build an automatic pipeline to collect data, with human verification to ensure quality. The final dataset contains 7,659 physical scenarios with 334 human-verified examples as the test set. We evaluated 10 contemporary LLMs and found that even the strongest model achieves only a 21.5% pass rate, demonstrating the difficulty of the task. Finally, we introduce a reinforcement learning pipeline with visual rewards that uses a vision-language model as a judge to train textual models. Experiments show that training with our data improves physical simulation via code while substantially enhancing general code generation performance.