Self-Steering Language Models
This addresses the challenge of efficient and reliable reasoning in language models for complex tasks, offering a novel approach to improve performance without fine-tuning.
The paper tackles the problem of slow and error-prone natural language reasoning in language models by introducing DisCIPL, a self-steering method where a Planner generates task-specific inference programs executed by Followers, enabling verifiable and efficient reasoning. The result shows that DisCIPL with small models like Llama-3.2-1B matches or outperforms larger models such as GPT-4o on constrained generation tasks.
While test-time reasoning enables language models (LMs) to tackle complex tasks, searching or planning in natural language can be slow, costly, and error-prone. But even when LMs struggle to emulate the precise reasoning steps needed to solve a problem, they often excel at describing its abstract structure--both how to verify solutions and how to search for them. This paper introduces DisCIPL, a method for "self-steering" LMs where a Planner model generates a task-specific inference program that is executed by a population of Follower models. Our approach equips LMs with the ability to write recursive search procedures that guide LM inference, enabling new forms of verifiable and efficient reasoning. When instantiated with a small Follower (e.g., Llama-3.2-1B or Qwen3-1.7B), DisCIPL matches (and sometimes outperforms) much larger models, including GPT-4o and o1, on challenging constrained generation tasks. Our work opens up a design space of highly-parallelized Monte Carlo inference strategies that outperform standard best-of-N sampling, require no finetuning, and can be implemented automatically by existing LMs.