NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
This addresses the issue of imperfect visual perception affecting reasoning in VLMs, though it is incremental as it builds on existing RL methods with a simple augmentation approach.
The paper tackles the problem of enhancing policy exploration and robustness in vision-language models for visual reasoning by introducing NoisyRollout, a data augmentation method that mixes clean and distorted image trajectories, achieving state-of-the-art performance on 5 out-of-domain benchmarks.
Recent advances in reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy exploration to better scale test-time compute remains largely underexplored. In addition, VLMs continue to struggle with imperfect visual perception, which in turn affects the subsequent reasoning process. We introduce NoisyRollout, a simple yet effective data augmentation method that addresses these issues by mixing training trajectories from both clean and moderately distorted images. This approach injects perceptual diversity, encouraging better policy exploration and leading to more robust reasoning. A noise annealing schedule gradually reduces distortion strength, aiding exploration early in training while ensuring later stability. Crucially, our method is easy-to-adopt--requiring no additional training cost and no modifications to the RL objective. Extensive experiments on 2 distinct training datasets demonstrate that NoisyRollout achieves state-of-the-art performance among open-source RL-tuned models across 5 out-of-domain reasoning and perception benchmarks. Furthermore, we validate the effectiveness of NoisyRollout across model sizes (7B and 32B), data scales (from 1K to 6K) and image augmentation types (Gaussion noise and rotation), highlighting its generalizability and scalability.