Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?
This work provides a roadmap for improving language models in mathematical reasoning, though it is incremental in understanding SFT's limitations.
The paper analyzed how supervised fine-tuning (SFT) affects language models' performance on mathematical reasoning tasks, finding that models can progress from easy to medium difficulty with minimal SFT but plateau at 65% accuracy on hard questions and struggle with extremely hard ones requiring unconventional skills.
Recent supervised fine-tuning (SFT) approaches have significantly improved language models' performance on mathematical reasoning tasks, even when models are trained at a small scale. However, the specific capabilities enhanced through such fine-tuning remain poorly understood. In this paper, we conduct a detailed analysis of model performance on the AIME24 dataset to understand how reasoning capabilities evolve. We discover a ladder-like structure in problem difficulty, categorize questions into four tiers (Easy, Medium, Hard, and Extremely Hard (Exh)), and identify the specific requirements for advancing between tiers. We find that progression from Easy to Medium tier requires adopting an R1 reasoning style with minimal SFT (500-1K instances), while Hard-level questions suffer from frequent model's errors at each step of the reasoning chain, with accuracy plateauing at around 65% despite logarithmic scaling. Exh-level questions present a fundamentally different challenge; they require unconventional problem-solving skills that current models uniformly struggle with. Additional findings reveal that carefully curated small-scale datasets offer limited advantage-scaling dataset size proves far more effective. Our analysis provides a clearer roadmap for advancing language model capabilities in mathematical reasoning.