When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics
For practitioners deploying VLMs, STS offers a more effective token pruning method that preserves structural and semantic information, reducing computational overhead while maintaining accuracy.
Vision-Language Models suffer from high computational cost; existing token pruning methods collapse attention on similar regions, losing diversity. STS introduces a two-stage pruning framework (structure then semantics) that improves both diversity and task alignment, achieving better efficiency without sacrificing performance.
Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores. This single-metric paradigm presents a critical flaw: high attention scores inherently collapse onto semantically similar regions, thereby severely reducing feature diversity and discarding vital contextual details. To address this, we introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework that explicitly decouples the pruning process. The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity. The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens. This two-stage synergy constitutes the core of STS, first ensuring geometric coverage and then refining the retained tokens according to semantic relevance. Extensive evaluations demonstrate that STS mitigates the redundancy caused by attention-based selection, improving both structural diversity and fine-grained task alignment of the preserved visual tokens.