Didactic to Constructive: Turning Expert Solutions into Learnable Reasoning
This addresses the challenge of training LLMs on difficult problems where expert solutions are expensive and not directly usable, offering a sample-efficient method for enhancing reasoning capabilities.
The paper tackles the problem of improving reasoning in large language models by leveraging expert human solutions, which are often didactic and out-of-distribution for models, and proposes Distribution Aligned Imitation Learning (DAIL) to transform these solutions into learnable reasoning traces, achieving 10-25% pass@k gains, 2x to 4x reasoning efficiency improvements, and out-of-domain generalization with fewer than 1000 expert solutions.
Improving the reasoning capabilities of large language models (LLMs) typically relies either on the model's ability to sample a correct solution to be reinforced or on the existence of a stronger model able to solve the problem. However, many difficult problems remain intractable for even current frontier models, preventing the extraction of valid training signals. A promising alternative is to leverage high-quality expert human solutions, yet naive imitation of this data fails because it is fundamentally out of distribution: expert solutions are typically didactic, containing implicit reasoning gaps intended for human readers rather than computational models. Furthermore, high-quality expert solutions are expensive, necessitating generalizable sample-efficient training methods. We propose Distribution Aligned Imitation Learning (DAIL), a two-step method that bridges the distributional gap by first transforming expert solutions into detailed, in-distribution reasoning traces and then applying a contrastive objective to focus learning on expert insights and methodologies. We find that DAIL can leverage fewer than 1000 high-quality expert solutions to achieve 10-25% pass@k gains on Qwen2.5-Instruct and Qwen3 models, improve reasoning efficiency by 2x to 4x, and enable out-of-domain generalization.