GAPO: Robust Advantage Estimation for Real-World Code LLMs
This addresses robustness issues in RL for code-editing LLMs, though it appears incremental as an enhancement to existing group-relative methods.
The paper tackles the problem of skewed reward distributions with outliers in reinforcement learning for code-editing LLMs, proposing GAPO which improves exact match accuracy over existing methods like GRPO and DAPO across nine models on 51,844 real-world tasks.
Reinforcement learning (RL) is widely used for post-training large language models (LLMs) in code editing, where group-relative methods like GRPO are popular for their critic-free, normalized advantage estimation. However, in real-world code-editing scenarios, reward distributions are often skewed with unpredictable outliers, leading to distorted advantage computation and increased noise. To address this issue, we propose Group Adaptive Policy Optimization (GAPO), which adaptively finds an outlier-free highest-density interval (HDI) per prompt and then uses the median of that interval as an adaptive Q to replace the group mean in advantage calculation. This adaptive Q robustly handles skewed distributions while remaining plug-and-play and efficient. We validate GAPO on nine instruction-tuned LLMs (3B-14B) using a large internal dataset of 51,844 real-world, history-aware code-editing tasks across 10 languages, demonstrating consistent improvements in exact match accuracy over GRPO and its variant DAPO. Code is publicly available.