DRIP: Dynamic patch Reduction via Interpretable Pooling
This work addresses efficiency concerns for researchers and practitioners in multimodal AI, though it is incremental as it builds on existing vision-language models.
The paper tackles the efficiency problem of vision-language models by proposing DRIP, a method that dynamically merges tokens in deeper layers of visual encoders, achieving significant GFLOP reduction while maintaining comparable performance in classification and zero-shot tasks.
Recently, the advances in vision-language models, including contrastive pretraining and instruction tuning, have greatly pushed the frontier of multimodal AI. However, owing to the large-scale and hence expensive pretraining, the efficiency concern has discouraged researchers from attempting to pretrain a vision language model from scratch. In this work, we propose Dynamic patch Reduction via Interpretable Pooling (DRIP), which adapts to the input images and dynamically merges tokens in the deeper layers of a visual encoder. Our results on both ImageNet training from scratch and CLIP contrastive pretraining demonstrate a significant GFLOP reduction while maintaining comparable classification/zero-shot performance. To further validate our proposed method, we conduct continual pretraining on a large biology dataset, extending its impact into scientific domains.