Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
This addresses a key limitation in LLMs for users needing diverse outputs, though it is incremental as it builds on existing methods like GRPO.
The paper tackles the problem of mode collapse in Large Language Models (LLMs), where models generate repetitive completions, by introducing Group-Aware Policy Optimization (GAPO), which improves response diversity without compromising accuracy on benchmarks like GSM8K, MATH, HumanEval, and MMLU-Pro.
Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce Group-Aware Policy Optimization (GAPO), a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage. We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses. Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available.