Instruction Anchors: Dissecting the Causal Dynamics of Modality Arbitration
This work addresses the problem of understanding and controlling modality arbitration in MLLMs for improved safety and reliability, though it is incremental in dissecting existing mechanisms.
The paper investigates how multimodal large language models (MLLMs) decide to use multimodal contexts based on user instructions, finding that instruction tokens act as anchors for modality arbitration, with shallow attention layers routing information and deep layers resolving competition. Causal interventions show that manipulating 5% of critical attention heads can decrease or increase the modality-following ratio by 60%.
Modality following serves as the capacity of multimodal large language models (MLLMs) to selectively utilize multimodal contexts based on user instructions. It is fundamental to ensuring safety and reliability in real-world deployments. However, the underlying mechanisms governing this decision-making process remain poorly understood. In this paper, we investigate its working mechanism through an information flow lens. Our findings reveal that instruction tokens function as structural anchors for modality arbitration: Shallow attention layers perform non-selective information transfer, routing multimodal cues to these anchors as a latent buffer; Modality competition is resolved within deep attention layers guided by the instruction intent, while MLP layers exhibit semantic inertia, acting as an adversarial force. Furthermore, we identify a sparse set of specialized attention heads that drive this arbitration. Causal interventions demonstrate that manipulating a mere $5\%$ of these critical heads can decrease the modality-following ratio by $60\%$ through blocking, or increase it by $60\%$ through targeted amplification of failed samples. Our work provides a substantial step toward model transparency and offers a principled framework for the orchestration of multimodal information in MLLMs.