AMPS: Adaptive Modality Preference Steering via Functional Entropy
This work addresses modality preference issues in multimodal large language models, which is an incremental improvement for enhancing model robustness in multimodal tasks.
The paper tackles the problem of multimodal large language models exhibiting modality preference, where they over-rely on one modality over another, by introducing an instance-aware diagnostic metric and scaling strategy to adjust preference adaptively. The result shows that this approach outperforms conventional uniform steering in modulating modality preference while keeping generation error rates low.
Multimodal Large Language Models (MLLMs) often exhibit significant modality preference, which is a tendency to favor one modality over another. Depending on the input, they may over-rely on linguistic priors relative to visual evidence, or conversely over-attend to visually salient but facts in textual contexts. Prior work has applied a uniform steering intensity to adjust the modality preference of MLLMs. However, strong steering can impair standard inference and increase error rates, whereas weak steering is often ineffective. In addition, because steering sensitivity varies substantially across multimodal instances, a single global strength is difficult to calibrate. To address this limitation with minimal disruption to inference, we introduce an instance-aware diagnostic metric that quantifies each modality's information contribution and reveals sample-specific susceptibility to steering. Building on these insights, we propose a scaling strategy that reduces steering for sensitive samples and a learnable module that infers scaling patterns, enabling instance-aware control of modality preference. Experimental results show that our instance-aware steering outperforms conventional steering in modulating modality preference, achieving effective adjustment while keeping generation error rates low.