ABC: Achieving Better Control of Multimodal Embeddings using VLMs
This addresses the need for better user control in visual embeddings for tasks with ambiguity, though it appears incremental as it builds on existing vision-language model backbones.
The paper tackles the problem of weak user control in visual embedding models by introducing ABC, a multimodal embedding model that integrates image features with natural language instructions, achieving best-for-size performance on MSCOCO image-to-text retrieval and top results on classification and VQA tasks in the Massive Multimodal Embedding Benchmark.
Visual embedding models excel at zero-shot tasks like visual retrieval and classification. However, these models cannot be used for tasks that contain ambiguity or require user instruction. These tasks necessitate an embedding model which outputs can use a natural language instruction to control the representation of a visual embedding. Existing CLIP-based approaches embed images and text independently, and fuse the result. We find that this results in weak interactions between modalities, and poor user control over the representation. We introduce ABC, an open-source multimodal embedding model that uses a vision-language model backbone to deeply integrate image features with natural language instructions. ABC achieves best-for-size performance on MSCOCO image-to-text retrieval and is the top performing model on classification and VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly unified vision-language representation, ABC can use natural language to solve subtle and potentially ambiguous visual retrieval problems. To evaluate this capability, we design CtrlBench, a benchmark that requires interleaving textual instructions with image content for correct retrieval. ABC advances the state of visual embeddings, outputting high-quality visual representations with natural language control. Our model and datasets are available at our project page: https://tiger-ai-lab.github.io/ABC/