Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
This addresses the need for general-purpose, reliable reward models for multimodal alignment, representing a strong specific gain in the domain of vision-language models.
The paper tackles the problem of providing reward signals for multimodal understanding and reasoning tasks by proposing Skywork-VL Reward, a reward model that achieves state-of-the-art results on multimodal VL-RewardBench and competitive performance on text-only RewardBench.
We propose Skywork-VL Reward, a multimodal reward model that provides reward signals for both multimodal understanding and reasoning tasks. Our technical approach comprises two key components: First, we construct a large-scale multimodal preference dataset that covers a wide range of tasks and scenarios, with responses collected from both standard vision-language models (VLMs) and advanced VLM reasoners. Second, we design a reward model architecture based on Qwen2.5-VL-7B-Instruct, integrating a reward head and applying multi-stage fine-tuning using pairwise ranking loss on pairwise preference data. Experimental evaluations show that Skywork-VL Reward achieves state-of-the-art results on multimodal VL-RewardBench and exhibits competitive performance on the text-only RewardBench benchmark. Furthermore, preference data constructed based on our Skywork-VL Reward proves highly effective for training Mixed Preference Optimization (MPO), leading to significant improvements in multimodal reasoning capabilities. Our results underscore Skywork-VL Reward as a significant advancement toward general-purpose, reliable reward models for multimodal alignment. Our model has been publicly released to promote transparency and reproducibility.