Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
This work addresses measurement reliability and cost transparency in RLVR for large language models, highlighting incremental improvements in evaluation practices.
The paper investigates the overstated gains in reinforcement learning with verifiable rewards (RLVR) for tasks like math and code, showing that under parity-controlled evaluation, many reported improvements shrink or vanish due to issues like data contamination and evaluation pitfalls. It proposes a tax-aware protocol that revises prior conclusions and provides more reliable estimates of reasoning gains.
Reinforcement learning with verifiable rewards (RLVR) is a practical and scalable approach to enhancing large language models in areas such as math, code, and other structured tasks. Two questions motivate this paper: how much of the reported gains survive under strictly parity-controlled evaluation, and whether RLVR is cost-free or exacts a measurable tax. We argue that progress is real, but gains are often overstated due to three forces - an RLVR tax, evaluation pitfalls, and data contamination. Using a partial-prompt contamination audit and matched-budget reproductions across base and RL models, we show that several headline gaps shrink or vanish under clean, parity-controlled evaluation. We then propose a tax-aware training and evaluation protocol that co-optimizes accuracy, grounding, and calibrated abstention and standardizes budgeting and provenance checks. Applied to recent RLVR setups, this protocol yields more reliable estimates of reasoning gains and, in several cases, revises prior conclusions. Our position is constructive: RLVR is valuable and industry-ready; we advocate keeping its practical benefits while prioritizing reliability, safety, and measurement.