Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
This work addresses a key problem in stabilizing reinforcement learning fine-tuning for AI applications like dialogue and code generation, offering an incremental improvement over existing methods.
The paper tackles the conflict between constraints and optimization in reinforcement learning fine-tuning by introducing dynamic constraints that adapt based on output quality, resulting in higher task rewards and improved stability in dialogue and code generation tasks.
Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose \textit{dynamic constraints} that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an \textit{online refiner} that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability.