CRISP: Complex Reasoning with Interpretable Step-based Plans
This addresses the need for improved reasoning in AI systems for domains like math and coding, though it is incremental as it builds on existing plan generation approaches.
The authors tackled the problem of insufficient reasoning capabilities in large language models for complex tasks by introducing CRISP, a dataset of high-level plans for mathematical reasoning and code generation, and showed that fine-tuning a small model on it outperforms larger models using few-shot prompting and Chain-of-Thought reasoning.
Recent advancements in large language models (LLMs) underscore the need for stronger reasoning capabilities to solve complex problems effectively. While Chain-of-Thought (CoT) reasoning has been a step forward, it remains insufficient for many domains. A promising alternative is explicit high-level plan generation, but existing approaches largely assume that LLMs can produce effective plans through few-shot prompting alone, without additional training. In this work, we challenge this assumption and introduce CRISP (Complex Reasoning with Interpretable Step-based Plans), a multi-domain dataset of high-level plans for mathematical reasoning and code generation. The plans in CRISP are automatically generated and rigorously validated--both intrinsically, using an LLM as a judge, and extrinsically, by evaluating their impact on downstream task performance. We demonstrate that fine-tuning a small model on CRISP enables it to generate higher-quality plans than much larger models using few-shot prompting, while significantly outperforming Chain-of-Thought reasoning. Furthermore, our out-of-domain evaluation reveals that fine-tuning on one domain improves plan generation in the other, highlighting the generalizability of learned planning capabilities.