When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
For researchers working on large vision-language models, this work provides a unified framework to understand and mitigate the harmful effects of attention sinks, improving model performance on multimodal tasks.
The paper categorizes attention sinks in LVLMs into V-sinks and L-sinks, revealing a trade-off between global priors and local perception. It proposes LSG, a lightweight module that improves multimodal benchmarks by dynamically modulating sink contributions.
Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.