Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners
For researchers studying neural relational reasoning, this work provides an automated way to generate challenging benchmarks, potentially accelerating progress in systematic generalization.
The paper addresses the challenge of evaluating neural relational reasoners on harder instances than seen in training by using LLMs to automate benchmark generation. It shows that LLM-driven evolutionary search can produce hard instances that improve the Edge Transformer's generalization, and that the method extends to novel worlds proposed by LLMs.
Reasoning about relational structures remains a significant challenge for neural models, particularly when they must systematically apply learned knowledge to problem instances that are harder than those seen in training. Progress is hampered by the difficulty of evaluating such generalization, since a priori, it is rarely clear what makes an instance hard. We study how this issue can be addressed by using large language models (LLMs) to automate benchmark generation, learning to produce increasingly challenging instances in an end-to-end manner. Concretely, given a world parametrized by Datalog rules, and an Edge Transformer as the reasoning evaluator, we use LLM-driven evolutionary search (based on FunSearch) and autonomous agentic search to discover sampling functions that yield hard problem instances. We also show that the Edge Transformer can be improved using this data such that it generalizes well to further data perturbations. Finally, we show that the same machinery can be applied to novel worlds proposed by LLMs, opening the door to autonomous research on neural relational reasoning.