SPhyR: Spatial-Physical Reasoning Benchmark on Material Distribution
This provides a domain-specific benchmark for evaluating spatial and physical reasoning in LLMs, complementing existing language and logic benchmarks.
The authors introduced SPhyR, a novel dataset for benchmarking spatial-physical reasoning in Large Language Models based on topology optimization problems, where models must predict optimal material distributions from given loads and supports without simulation tools. The dataset includes tasks like filling masked regions and predicting complete distributions, requiring understanding of force flow and structural stability.
We introduce a novel dataset designed to benchmark the physical and spatial reasoning capabilities of Large Language Models (LLM) based on topology optimization, a method for computing optimal material distributions within a design space under prescribed loads and supports. In this dataset, LLMs are provided with conditions such as 2D boundary, applied forces and supports, and must reason about the resulting optimal material distribution. The dataset includes a variety of tasks, ranging from filling in masked regions within partial structures to predicting complete material distributions. Solving these tasks requires understanding the flow of forces and the required material distribution under given constraints, without access to simulation tools or explicit physical models, challenging models to reason about structural stability and spatial organization. Our dataset targets the evaluation of spatial and physical reasoning abilities in 2D settings, offering a complementary perspective to traditional language and logic benchmarks.