t-gems: text-guided exit modules for decreasing clip image encoder
For practitioners deploying multimodal models, this work offers a practical method to reduce inference cost without retraining, though it is incremental over existing early exit techniques.
T-GEMs reduce CLIP image encoder computational cost by enabling early exits guided by text, achieving up to 50% reduction in FLOPs with minimal accuracy loss on retrieval tasks.
Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance.