f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment
This work provides a unified framework and new algorithms for general LLM alignment, which is a critical problem for developers and users of LLMs.
This paper tackles the problem of aligning large language models (LLMs) using divergence-based reinforcement learning. They propose f-GRPO and f-HAL, which are theoretically guaranteed to improve average reward and empirically demonstrate superior performance on math reasoning and safety alignment tasks.
Recent research shows that Preference Alignment (PA) objectives act as divergence estimators between aligned (chosen) and unaligned (rejected) response distributions. In this work, we extend this divergence-based perspective to general alignment settings, such as reinforcement learning with verifiable rewards (RLVR), where only environmental rewards are available. Within this unified framework, we propose f-Group Relative Policy Optimization (f-GRPO), a class of on-policy reinforcement learning, and f-Hybrid Alignment Loss (f-HAL), a hybrid on/off policy objectives, for general LLM alignment based on variational representation of f-divergences. We provide theoretical guarantees that these classes of objectives improve the average reward after alignment. Empirically, we validate our framework on both RLVR (Math Reasoning) and PA tasks (Safety Alignment), demonstrating superior performance and flexibility compared to current methods.