MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop
This addresses the issue of uninformative feedback in RLVR for reasoning tasks, offering an incremental improvement by incorporating multi-turn verbal feedback.
The paper tackles the problem of sparse and uninformative scalar rewards in Reinforcement Learning with Verifiable Rewards (RLVR) by leveraging verbal feedback to guide training on failed samples, resulting in outperformance of supervised fine-tuning and RLVR baselines in-domain and good generalization out-of-domain.
Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in multiple domains, yet outcome-only scalar rewards are often sparse and uninformative, especially on failed samples, where they merely indicate failure and provide no insight into why the reasoning fails. In this paper, we investigate how to leverage richer verbal feedback to guide RLVR training on failed samples, and how to convert such feedback into a trainable learning signal. Specifically, we propose a multi-turn feedback-guided reinforcement learning framework. It builds on three mechanisms: (1) dynamic multi-turn regeneration guided by feedback, triggered only on failed samples, (2) two complementary learning signals for within-turn and cross-turn optimization, and (3) structured feedback injection into the model's reasoning process. Trained on sampled OpenR1-Math, the approach outperforms supervised fine-tuning and RLVR baselines in-domain and generalizes well out-of-domain.