Evaluating and Steering Modality Preferences in Multimodal Large Language Model
This work addresses the problem of modality bias in MLLMs for researchers and developers, offering a method to control preferences without fine-tuning, though it is incremental as it builds on existing representation engineering techniques.
The study investigated modality preference in multimodal large language models (MLLMs) by creating the MC² benchmark to evaluate bias in conflicting evidence scenarios, finding that all 18 tested models exhibited clear modality bias, and proposed a representation engineering method to steer this preference, achieving improvements in tasks like hallucination mitigation and multimodal machine translation.
Multimodal large language models (MLLMs) have achieved remarkable performance on complex tasks with multimodal context. However, it is still understudied whether they exhibit modality preference when processing multimodal contexts. To study this question, we first build a \textbf{MC\textsuperscript{2}} benchmark under controlled evidence conflict scenarios to systematically evaluate modality preference, which is the tendency to favor one modality over another when making decisions based on multimodal conflicting evidence. Our extensive evaluation reveals that all 18 tested MLLMs generally demonstrate clear modality bias, and modality preference can be influenced by external interventions. An in-depth analysis reveals that the preference direction can be captured within the latent representations of MLLMs. Built on this, we propose a probing and steering method based on representation engineering to explicitly control modality preference without additional fine-tuning or carefully crafted prompts. Our method effectively amplifies modality preference toward a desired direction and applies to downstream tasks such as hallucination mitigation and multimodal machine translation, yielding promising improvements.