DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
This provides a rigorous, provably correct dataset for fine-grained probing of compositional reasoning abilities in LLMs, addressing a key bottleneck in AI evaluation.
The authors tackled the problem of evaluating compositional spatial reasoning in Large Language Models by introducing DecompSR, a large benchmark dataset with over 5 million datapoints, and found that LLMs struggle with productive and systematic generalization while being more robust to linguistic variation.
We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning ability. The generation of DecompSR allows users to independently vary several aspects of compositionality, namely: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order, distractors) and systematicity (novel linguistic elements). DecompSR is built procedurally in a manner which makes it is correct by construction, which is independently verified using a symbolic solver to guarantee the correctness of the dataset. DecompSR is comprehensively benchmarked across a host of Large Language Models (LLMs) where we show that LLMs struggle with productive and systematic generalisation in spatial reasoning tasks whereas they are more robust to linguistic variation. DecompSR provides a provably correct and rigorous benchmarking dataset with a novel ability to independently vary the degrees of several key aspects of compositionality, allowing for robust and fine-grained probing of the compositional reasoning abilities of LLMs.