SVD-Prune: Training-Free Token Pruning For Efficient Vision-Language Models
For practitioners deploying VLMs with limited computational resources, this method offers a simple, plug-and-play solution to reduce vision token count without retraining, achieving better retention of visual information than existing heuristics.
SVD-Prune introduces a training-free token pruning method for vision-language models that uses singular value decomposition and leverage scores to preserve tokens contributing most to global variance, outperforming prior methods and maintaining strong performance with as few as 16-32 vision tokens.
Vision-Language Models (VLM) have revolutionized multimodal learning by jointly processing visual and textual information. Yet, they face significant challenges due to the high computational and memory demands of processing long sequences of vision tokens. Many existing methods rely on local heuristics, such as attention scores or token norms. However, these criteria suffer from positional bias and information dispersion, limiting their ability to preserve essential content at high pruning ratios and leading to performance degradation on visually detailed images. To address these issues, we propose SVD-Prune, a trainingfree, plug-and-play token pruning method based on Singular Value Decomposition. It decomposes the vision token feature matrix and selects the top-K tokens using statistical leverage scores, ensuring only tokens contributing most to the dominant global variance are preserved. Experiments show that SVD-Prune consistently outperforms prior pruning methods under extreme vision token budgets, maintaining strong performance even with 32 and 16 vision tokens.