ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL
This work addresses the problem of enhancing knowledge discovery efficiency for MLLMs, offering a method to achieve higher performance with smaller models.
This paper introduces ContextRL, a framework that uses context augmentation to improve knowledge discovery efficiency in MLLMs. It enables the Qwen3-VL-8B model to achieve performance comparable to a 32B model, significantly outperforming standard RLVR baselines.
We propose ContextRL, a novel framework that leverages context augmentation to overcome these bottlenecks. Specifically, to enhance Identifiability, we provide the reward model with full reference solutions as context, enabling fine-grained process verification to filter out false positives (samples with the right answer but low-quality reasoning process). To improve Reachability, we introduce a multi-turn sampling strategy where the reward model generates mistake reports for failed attempts, guiding the policy to "recover" correct responses from previously all-negative groups. Experimental results on 11 perception and reasoning benchmarks show that ContextRL significantly improves knowledge discovery efficiency. Notably, ContextRL enables the Qwen3-VL-8B model to achieve performance comparable to the 32B model, outperforming standard RLVR baselines by a large margin while effectively mitigating reward hacking. Our in-depth analysis reveals the significant potential of contextual information for improving reward model accuracy and document the widespread occurrence of reward hacking, offering valuable insights for future RLVR research.