Before the Model Learns the Bug:Fuzzing RLVR Verifiers
For practitioners using RLVR with automated reward functions, this work addresses the overlooked risk of verifier bugs being learned and exploited, providing a practical detection method.
The paper identifies that bugs in verifiers used for reinforcement learning with verifiable rewards (RLVR) can be exploited by optimization, leading to incorrect learning. It introduces a lightweight fuzzing framework to detect such bugs by generating adversarial completions and comparing verifier outputs.
Reinforcement learning with verifiable rewards (RLVR) replaces human preference labels with executable reward functions such as math answer checkers, JSON tool-call validators, and code unit-test harnesses. That makes the reward partly a software artifact: if the verifier is wrong, optimization can learn the bug. We study this failure mode with a lightweight verifier-fuzzing framework that generates adversarial completions, compares buggy and stricter reference verifiers, logs paired decisions, and reports false-positive, false-negative, disagreement, exploit, and uncertainty metrics.