SLR: Automated Synthesis for Scalable Logical Reasoning
This work addresses the challenge of scalable and precise logical reasoning evaluation for LLM developers, though it is incremental as it builds on existing reasoning methods.
The authors tackled the problem of evaluating and training Large Language Models (LLMs) for logical reasoning by introducing SLR, an automated framework that synthesizes prompts, validation programs, and ground-truth rules without human annotations, resulting in curriculum learning that doubled Llama-3-8B accuracy on their benchmark and achieved parity with Gemini-Flash-Thinking at lower computational cost.
We introduce SLR, an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via Scalable Logical Reasoning. Given a user's task specification, SLR automatically synthesizes (i) an instruction prompt for an inductive reasoning task, (ii) a validation program, executable on model outputs to provide verifiable rewards, and (iii) the latent ground-truth rule. This process is fully automated, scalable, requires no human annotations, and offers precise control over task difficulty. Using SLR, we create SLR-Bench, a benchmark comprising 19k prompts organized into 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs demonstrate improved performance but incur very high test-time computation, with costs exceeding $300 for just 1,000 prompts. Finally, curriculum learning via SLR doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. Moreover, these reasoning capabilities generalize to a wide range of established benchmarks, underscoring the effectiveness of SLR for downstream reasoning.