Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?
This addresses the problem of error propagation in mathematical reasoning for LLM users, though it appears incremental as it builds on existing CoT and RL methods.
The researchers tackled the problem of large language models being brittle to early errors in mathematical reasoning by investigating whether training on intentionally flawed reasoning traces could teach models to detect and recover from errors without degrading standard problem-solving ability. Their Mixed-CoT-RL model matched standard RL on clean problems (41% vs 41%) while substantially outperforming it on problems with flawed reasoning (24% vs 19%).
Chain-of-thought (CoT) prompting has become central to mathematical reasoning in large language models, yet models remain brittle to early errors: a single arithmetic slip or unjustified inference typically propagates uncorrected to an incorrect final answer. We investigate whether training on intentionally flawed reasoning traces can teach models to detect and recover from such errors without degrading standard problem-solving ability. Using competition-level problems from MATH-lighteval, we generate CoT prefixes containing exactly one controlled error, either a calculation error (sign flips, dropped terms) or a reasoning error (misapplied rules, unjustified logical steps), and fine-tune Qwen3-4B with GRPO using a binary final-answer reward. Our Mixed-CoT-RL model matches standard RL on clean problems (41% vs 41%) while substantially outperforming it on problems prefilled with flawed reasoning (24% vs 19%). Notably, clean-only RL fine-tuning degrades robustness below the untuned baseline 19% vs. 20%), indicating that conventional training increases susceptibility to misleading prefills. Among error types, training on reasoning errors yields greater robustness gains than calculation errors alone, with mixed training performing best. These findings demonstrate that exposure to flawed traces during training can improve error-recovery behavior without sacrificing accuracy, suggesting a path toward more robust mathematical reasoning in LLMs.