Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
This addresses efficiency issues for users of VLMs, offering a practical improvement over existing token pruning methods.
The paper tackles the problem of high inference costs in Vision-Language Models (VLMs) due to redundant visual tokens by proposing a training-free token pruning method that uses zeroth-order gradient estimation to identify influential tokens. The result is pruning up to 94.4% of tokens while maintaining accuracy and achieving up to 2.30x faster inference.
Large Vision-Language Models (VLMs) enable strong multimodal reasoning but incur heavy inference costs from redundant visual tokens. Token pruning alleviates this issue, yet existing approaches face limitations. Attention-based methods rely on raw attention scores, which are often unstable across layers and heads and can lead to redundant selections. Diversity-based methods improve robustness by selecting tokens far apart in feature space but risk dropping regions needed for accurate prediction. We propose \ours, a training-free framework built on a simple intuition: tokens with higher sensitivity are more likely to influence the model's output, and they should also capture complementary visual cues rather than overlapping information. To achieve this, we estimate token sensitivity using zeroth-order perturbations at the projection layer, a shallow and computationally light component of the model. This approach measures how small random perturbations affect the projection outputs, allowing us to approximate each token's influence through lightweight forward passes without backpropagation. Extensive experiments across multiple VLMs and benchmarks show that \ours consistently outperforms prior methods, pruning up to 94.4\% of tokens while maintaining accuracy and significantly improving efficiency, achieving up to 2.30x faster end-to-end inference over the baseline.