VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
This work addresses efficiency issues for researchers and practitioners using ViT-based segmentation models, particularly in task-oriented scenarios, by providing a novel pruning method that maintains performance while reducing computational overhead.
The paper tackles the high computational cost of Vision Transformers in task-oriented segmentation by introducing VLTP, a token pruning mechanism guided by vision-language inputs, which reduces computational costs by approximately 25% without performance loss and by around 40% with only a 1% performance drop.
Vision Transformers (ViTs) have emerged as the backbone of many segmentation models, consistently achieving state-of-the-art (SOTA) performance. However, their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However, previous approaches fall short when applied to more complex task-oriented segmentation (TOS), where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP), a novel token pruning mechanism that can accelerate ViT-based segmentation models, particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers -- only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop. The code associated with this study can be found at this URL.