ASPO: Asymmetric Importance Sampling Policy Optimization
This addresses a fundamental flaw in LLM post-training for researchers and practitioners, though it is an incremental improvement over existing methods.
The paper tackled the problem of mismatched Importance Sampling (IS) ratios in token-level clipping for Outcome-Supervised Reinforcement Learning (OSRL) with Large Language Models, which caused unbalanced token weighting and suppressed updates. The result was that ASPO significantly mitigated premature convergence, improved training stability, and enhanced final performance over strong GRPO-based baselines on coding and mathematical reasoning benchmarks.
Recent Large Language Model (LLM) post-training methods rely on token-level clipping mechanisms during Reinforcement Learning (RL). However, we identify a fundamental flaw in this Outcome-Supervised RL (OSRL) paradigm: the Importance Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to unbalanced token weighting for positive and negative tokens. This mismatch suppresses the update of low-probability tokens while over-amplifying already high-probability ones. To address this, we propose Asymmetric Importance Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy that flips the IS ratios of positive-advantage tokens, aligning their update direction with the learning dynamics of negative ones. AIS further incorporates a soft dual-clipping mechanism to stabilize extreme updates while maintaining gradient flow. Comprehensive experiments on coding and mathematical reasoning benchmarks demonstrate that ASPO significantly mitigates premature convergence, improves training stability, and enhances final performance over strong GRPO-based baselines. Our analysis provides new insights into the role of token-level weighting in OSRL and highlights the critical importance of correcting IS in LLM RL. The code and models of ASPO are available at https://github.com/wizard-III/Archer2.0.