FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
This work addresses efficiency issues in vision-language models for researchers and practitioners, offering a novel pruning method that is incremental but provides strong specific gains.
The paper tackles the problem of high computational costs in large vision-language models caused by redundant vision tokens, proposing FlowCut, an information-flow-aware pruning framework that achieves superior results, including a 1.6% improvement on LLaVA-1.5-7B with 88.9% token reduction and a 4.3% improvement on LLaVA-NeXT-7B with 94.4% reduction, delivering a 3.2x speed-up.
Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises a basic question: Is such a simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts as an information relay, which can simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely on attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, an information-flow-aware pruning framework, mitigating the insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show that FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.5-7B with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2x speed-up in the prefilling stage. Our code is available at https://github.com/TungChintao/FlowCut