RL-finetuning LLMs from on- and off-policy data with a single algorithm
This work addresses the challenge of optimizing LLMs for specific tasks, offering a method that works with both on- and off-policy data, though it appears incremental as it builds on existing RL techniques.
The paper tackled the problem of fine-tuning large-language models using reinforcement learning by introducing AGRO, a novel algorithm based on generation consistency, and demonstrated its effectiveness with improved performance on a mathematical reasoning dataset over baselines.
We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms.