ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning
This addresses efficiency issues for users of large vision-language models, offering an incremental improvement over existing pruning methods.
The paper tackles the high computational cost of Large Vision-Language Models by proposing ViTCoP, a pruning framework that reduces redundancy in visual tokens, achieving state-of-the-art performance on image and video tasks while significantly lowering inference latency and GPU memory usage, with advantages especially evident at extreme pruning rates.
Large Vision-Language Models (LVLMs) incur high computational costs due to significant redundancy in their visual tokens. To effectively reduce this cost, researchers have proposed various visual token pruning methods. However, existing methods are generally limited, either losing critical visual information prematurely due to pruning in the vision encoder, or leading to information redundancy among the selected tokens due to pruning in the Large Language Models (LLMs). To address these challenges, we propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the LLM based on its hierarchical characteristics, to efficiently preserve critical and informationally diverse visual tokens. Meanwhile, to ensure compatibility with acceleration techniques like FlashAttention, we introduce the L2 norm of K-vectors as the token saliency metric in the LLM. Extensive experiments on various Large Vision-Language Models demonstrate that ViTCoP not only achieves state-of-the-art performance surpassing existing methods on both image and video understanding tasks, but also significantly reduces model inference latency and GPU memory consumption. Notably, its performance advantage over other methods becomes even more pronounced under extreme pruning rates.