Rethinking Causal Mask Attention for Vision-Language Inference
This addresses a specific bottleneck in vision-language models by modifying attention mechanisms to better handle multimodal inputs, representing an incremental improvement over existing approaches.
The paper tackled the problem that existing causal mask attention mechanisms from language models are suboptimal for vision-language models because they overly restrict access to future visual context, which contains important semantic information. The authors proposed a family of future-aware attention mechanisms that selectively compress future visual context into past representations, improving inference performance while maintaining autoregressive structure.
Causal attention has become a foundational mechanism in autoregressive vision-language models (VLMs), unifying textual and visual inputs under a single generative framework. However, existing causal mask-based strategies are inherited from large language models (LLMs) where they are tailored for text-only decoding, and their adaptation to vision tokens is insufficiently addressed in the prefill stage. Strictly masking future positions for vision queries introduces overly rigid constraints, which hinder the model's ability to leverage future context that often contains essential semantic cues for accurate inference. In this work, we empirically investigate how different causal masking strategies affect vision-language inference and then propose a family of future-aware attentions tailored for this setting. We first empirically analyze the effect of previewing future tokens for vision queries and demonstrate that rigid masking undermines the model's capacity to capture useful contextual semantic representations. Based on these findings, we propose a lightweight attention family that aggregates future visual context into past representations via pooling, effectively preserving the autoregressive structure while enhancing cross-token dependencies. We evaluate a range of causal masks across diverse vision-language inference settings and show that selectively compressing future semantic context into past representations benefits the inference.