Probing RLVR training instability through the lens of objective-level hacking
This addresses training instability issues for researchers and practitioners using RLVR in MoE models, offering guidance for stable algorithm design, but it is incremental as it builds on existing understanding of instability without introducing a new method to solve it.
The paper tackled the problem of training instability in reinforcement learning with verifiable rewards (RLVR) for large language models, particularly in Mixture-of-Experts (MoE) architectures, by introducing a framework based on objective-level hacking and tracing its mechanism to abnormal growth in training-inference discrepancy, with experiments on a 30B MoE model providing a causal explanation.
Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms.