Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
This work addresses a critical flaw in acceleration methods for vision-language models, particularly for vision-centric tasks, and is incremental as it builds on existing pruning approaches.
The paper tackled the problem of early visual token pruning in vision-language models, which maintains strong performance on many tasks but fails on vision-centric localization tasks, and proposed FEATHER, a method that resolves this issue and achieves over 5x performance improvement on localization benchmarks with comparable computational savings.
Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work, we examine the popular acceleration approach of early pruning of visual tokens inside the language model. Surprisingly, we find that while strong performance is maintained across many tasks, it exhibits drastically different behavior for a subset of vision-centric tasks such as localization. Upon further investigation, we uncover a core issue with the acceleration approach where most tokens towards the top of the image are pruned away. Yet, on many benchmarks aiming to evaluate vision-centric capabilities, strong performance persists with the flawed pruning strategy, highlighting these benchmarks' limited ability to assess fine-grained visual capabilities. Based on these findings, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble cRiteria), a straightforward approach that resolves the discovered early-layer pruning issue and further enhances the preservation of relevant tokens via multistage pruning with early uniform sampling to ensure broad image coverage. With comparable computational savings, we find that FEATHER achieves more than 5x performance improvement on the vision-centric localization benchmarks compared to the original acceleration approach.