HyperCLIP: Adapting Vision-Language models with Hypernetworks
This addresses the problem of efficient deployment of AI vision models for users with limited resources, though it is incremental as it builds on existing contrastive models.
The paper tackles the challenge of deploying large vision-language models in resource-constrained environments by proposing HyperCLIP, which uses a hypernetwork to adapt a small image encoder to text inputs, increasing zero-shot accuracy by up to 3% on ImageNet and 5% on CIFAR-100.
Self-supervised vision-language models trained with contrastive objectives form the basis of current state-of-the-art methods in AI vision tasks. The success of these models is a direct consequence of the huge web-scale datasets used to train them, but they require correspondingly large vision components to properly learn powerful and general representations from such a broad data domain. This poses a challenge for deploying large vision-language models, especially in resource-constrained environments. To address this, we propose an alternate vision-language architecture, called HyperCLIP, that uses a small image encoder along with a hypernetwork that dynamically adapts image encoder weights to each new set of text inputs. All three components of the model (hypernetwork, image encoder, and text encoder) are pre-trained jointly end-to-end, and with a trained HyperCLIP model, we can generate new zero-shot deployment-friendly image classifiers for any task with a single forward pass through the text encoder and hypernetwork. HyperCLIP increases the zero-shot accuracy of SigLIP trained models with small image encoders by up to 3% on ImageNet and 5% on CIFAR-100 with minimal training throughput overhead.