GRPO-MA: Multi-Answer Generation in GRPO for Stable and Efficient Chain-of-Thought Training
This work addresses efficiency and stability issues in reinforcement learning for reasoning in large models, offering an incremental improvement over existing GRPO methods.
The paper tackled challenges in GRPO for training Chain-of-Thought reasoning, including gradient coupling and sparse rewards, by proposing GRPO-MA, which uses multi-answer generation to reduce variance and improve stability, resulting in substantial performance gains on math, code, and multimodal tasks.
Recent progress, such as DeepSeek-R1, has shown that the GRPO algorithm, a Reinforcement Learning (RL) approach, can effectively train Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) and Vision-Language Models (VLMs). In this paper, we analyze three challenges of GRPO: gradient coupling between thoughts and answers, sparse reward signals caused by limited parallel sampling, and unstable advantage estimation. To mitigate these challenges, we propose GRPO-MA, a simple yet theoretically grounded method that leverages multi-answer generation from each thought process, enabling more robust and efficient optimization. Theoretically, we show that the variance of thought advantage decreases as the number of answers per thought increases. Empirically, our gradient analysis confirms this effect, showing that GRPO-MA reduces gradient spikes compared to GRPO. Experiments on math, code, and diverse multimodal tasks demonstrate that GRPO-MA substantially improves performance and training efficiency. Our ablation studies further reveal that increasing the number of answers per thought consistently enhances model performance.