Verifiable Accuracy and Abstention Rewards in Curriculum RL to Alleviate Lost-in-Conversation
This addresses the issue of unreliable multi-turn conversations in large language models, offering a practical solution for building more trustworthy systems, though it is incremental as it builds on existing reinforcement learning with verifiable rewards methods.
The paper tackles the problem of Lost-in-Conversation (LiC), where large language models degrade in multi-turn dialogues, by proposing a curriculum reinforcement learning framework that encourages correct answers and informed abstention. The result is a significant mitigation of LiC performance decay from 62.6% to 75.1% and improved abstention rates from 33.5% to 73.4% on benchmarks.
Large Language Models demonstrate strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC), a degradation in performance as information is revealed progressively in multi-turn settings. Motivated by the current progress on Reinforcement Learning with Verifiable Rewards (RLVR), we propose Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR), a framework that encourages models not only to generate correct answers, but also to judge the solvability of questions in the multi-turn conversation setting. Our approach employs a competence-gated curriculum that incrementally increases dialogue difficulty (in terms of instruction shards), stabilizing training while promoting reliability. Using multi-turn, on-policy rollouts and a mixed-reward system, RLAAR teaches models to balance problem-solving with informed abstention, reducing premature answering behaviors that cause LiC. Evaluated on LiC benchmarks, RLAAR significantly mitigates LiC performance decay (62.6% to 75.1%) and improves calibrated abstention rates (33.5% to 73.4%). Together, these results provide a practical recipe for building multi-turn reliable and trustworthy LLMs.