Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
For practitioners deploying VLMs on resource-constrained devices, this work offers a practical token reduction method that is compatible with modern acceleration frameworks, addressing a key bottleneck in efficient VLM inference.
The paper tackles the computational overhead of processing numerous visual tokens in Vision-Language Models (VLMs) for edge deployment. It proposes an attention-free, lightweight token reduction framework that achieves up to 2x speedup with less than 1% accuracy drop on multiple benchmarks.
Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.