Self-Supervised Open-Ended Classification with Small Visual Language Models
This addresses the challenge of few-shot open-ended classification for researchers and applications that lack access to large or proprietary models, offering a more efficient alternative.
The paper tackles the problem of enabling small visual language models to perform open-ended classification with few-shot learning by introducing Self-Context Adaptation (SeCAt), a self-supervised method that uses clustering and pseudocaptions to train models, resulting in outperforming larger models like Frozen and FROMAGe on multimodal few-shot datasets.
We present Self-Context Adaptation (SeCAt), a self-supervised approach that unlocks few-shot abilities for open-ended classification with small visual language models. Our approach imitates image captions in a self-supervised way based on clustering a large pool of images followed by assigning semantically-unrelated names to clusters. By doing so, we construct a training signal consisting of interleaved sequences of image and pseudocaption pairs and a query image, which we denote as the 'self-context' sequence. Based on this signal the model is trained to produce the right pseudo-caption. We demonstrate the performance and flexibility of SeCAt on several multimodal few-shot datasets, spanning various granularities. By using models with approximately 1B parameters we outperform the few-shot abilities of much larger models, such as Frozen and FROMAGe. SeCAt opens new possibilities for research and applications in open-ended few-shot learning that otherwise requires access to large or proprietary models.