The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
This addresses a key issue in aligning LLMs with human preferences for multimodal data, though it appears incremental as it builds on existing MM-RM frameworks.
The paper tackles the problem of multimodal reward models (MM-RMs) failing to generalize due to reliance on unimodal spurious correlations like text-only shortcuts, and introduces a Shortcut-aware learning algorithm that improves generalization, downstream task performance, and scalability.
Multimodal Reward Models (MM-RMs) are crucial for aligning Large Language Models (LLMs) with human preferences, particularly as LLMs increasingly interact with multimodal data. However, we find that MM-RMs trained on existing datasets often struggle to generalize to out-of-distribution data due to their reliance on unimodal spurious correlations, primarily text-only shortcuts within the training distribution, which prevents them from leveraging true multimodal reward functions. To address this, we introduce a Shortcut-aware MM-RM learning algorithm that mitigates this issue by dynamically reweighting training samples, shifting the distribution toward better multimodal understanding, and reducing dependence on unimodal spurious correlations. Our experiments demonstrate significant improvements in generalization, downstream task performance, and scalability, establishing a more robust framework for multimodal reward modeling.