Bridging Offline and Online Reinforcement Learning for LLMs
This addresses the problem of effectively transitioning RL methods for LLM finetuning, though it appears incremental in comparing existing objectives.
The study investigated reinforcement learning methods for finetuning large language models across offline, semi-online, and fully online regimes for verifiable and non-verifiable tasks, finding that online and semi-online methods strongly outperformed offline ones with similar performance between variants, and that multi-tasking with both reward types improved results.
We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types.