The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning
For practitioners distilling mathematical reasoning into small models, this reveals that high-reward data can be detrimental, challenging common assumptions and offering a practical fix.
The paper identifies a Quality-Utility Paradox in mathematical reasoning distillation: data refined by stronger models (higher reward scores) actually impairs small language models' performance compared to the SLM's own traces selected via rejection sampling. The authors propose Style-Aligned Refinement to mitigate this, restoring downstream utility.
Knowledge distillation from powerful reasoning models is widely used to improve Small Language Models (SLMs) on mathematical reasoning, often assuming that traces with higher reward model scores provide more useful supervision. We identify a counterintuitive \textbf{Quality-Utility Paradox} in mathematical reasoning distillation. Data refined or synthesized by a stronger Oracle obtains higher perceived quality according to reward models, yet consistently underperforms traces generated by the SLM itself and selected through rejection sampling across Qwen2.5, LLaMA-3, and DeepSeek families. Our analysis shows that Oracle refinement couples logical repair with distributional drift away from the SLM's native reasoning distribution. This drift increases the learner's adaptation cost and can outweigh the benefit of improved reasoning logic. To test this mechanism, we introduce \textbf{Style-Aligned Refinement}, which preserves the native trajectory of the SLM while retaining logical repair from the Oracle. This intervention lowers adaptation cost and restores downstream utility. These findings suggest that effective mathematical reasoning distillation should jointly optimize perceived solution quality and learner-data compatibility, rather than relying solely on reward-model scores. The datasets and code are available at https://github.com/Dracoqhl/Quality-Utility-Paradox.