SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models
This addresses efficiency issues for real-world applications of VLMs, representing an incremental improvement over existing methods.
The paper tackles the problem of redundancy in Transformer-based Vision-Language Models (VLMs) by proposing SmartTrim, an adaptive acceleration framework that prunes redundant tokens and attention heads, achieving 2-3 times faster inference with minimal performance degradation across various tasks.
Despite achieving remarkable performance on various vision-language tasks, Transformer-based Vision-Language Models (VLMs) suffer from redundancy in inputs and parameters, significantly hampering their efficiency in real-world applications. Moreover, the degree of redundancy in token representations and model parameters, such as attention heads, varies significantly for different inputs. In light of the challenges, we propose SmartTrim, an adaptive acceleration framework for VLMs, which adjusts the computational overhead per instance. Specifically, we integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. Furthermore, we devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart. Experimental results across various vision-language tasks consistently demonstrate that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation, highlighting the effectiveness and efficiency compared to previous approaches. Code will be available at https://github.com/kugwzk/SmartTrim.