Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models
This addresses the problem of expensive data and compute requirements for building advanced captioning models, offering a cost-effective solution for researchers and developers, though it is incremental in leveraging existing pretrained components.
The paper tackles the high cost of training vision-language models for captioning by introducing the Vision-Language-Vision auto-encoder framework, which distills knowledge from diffusion models and fine-tunes an LLM to achieve state-of-the-art captioning performance comparable to GPT-4o and Gemini 2.0 Flash while reducing training costs to under $1,000 USD.
Building state-of-the-art Vision-Language Models (VLMs) with strong captioning capabilities typically necessitates training on billions of high-quality image-text pairs, requiring millions of GPU hours. This paper introduces the Vision-Language-Vision (VLV) auto-encoder framework, which strategically leverages key pretrained components: a vision encoder, the decoder of a Text-to-Image (T2I) diffusion model, and subsequently, a Large Language Model (LLM). Specifically, we establish an information bottleneck by regularizing the language representation space, achieved through freezing the pretrained T2I diffusion decoder. Our VLV pipeline effectively distills knowledge from the text-conditioned diffusion model using continuous embeddings, demonstrating comprehensive semantic understanding via high-quality reconstructions. Furthermore, by fine-tuning a pretrained LLM to decode the intermediate language representations into detailed descriptions, we construct a state-of-the-art (SoTA) captioner comparable to leading models like GPT-4o and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and significantly reduces data requirements; by primarily utilizing single-modal images for training and maximizing the utility of existing pretrained models (image encoder, T2I diffusion model, and LLM), it circumvents the need for massive paired image-text datasets, keeping the total training expenditure under $1,000 USD.