Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning
This addresses the challenge of enhancing LLM reasoning for tasks like mathematical problem-solving, though it is incremental as it builds on existing negative sample methods.
The paper tackles the problem of improving Large Language Model reasoning by proposing Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples with correct format but incorrect answers, resulting in an average improvement of 2.03% over RL-trained models on mathematical reasoning benchmarks.
Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models.