RAVEN: Multitask Retrieval Augmented Vision-Language Learning
This work addresses the need for more efficient and accessible multimodal learning for researchers and practitioners, though it appears incremental as it builds on existing retrieval-augmented generation approaches.
The paper tackles the problem of applying retrieval-augmented generation to vision-language models, which is under-explored and limited by resource-intensive methods, by introducing RAVEN, a multitask framework that enhances base models through efficient fine-tuning without extra parameters, achieving improvements such as +1 CIDEr on MSCOCO and +3% accuracy on specific VQA question types.
The scaling of large language models to encode all the world's knowledge in model parameters is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. Existing methods focus on models designed for single tasks. Furthermore, they're limited by the need for resource intensive pre training, additional parameter requirements, unaddressed modality prioritization and lack of clear benefit over non-retrieval baselines. This paper introduces RAVEN, a multitask retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning. By integrating retrieval augmented samples without the need for additional retrieval-specific parameters, we show that the model acquires retrieval properties that are effective across multiple tasks. Our results and extensive ablations across retrieved modalities for the image captioning and VQA tasks indicate significant performance improvements compared to non retrieved baselines +1 CIDEr on MSCOCO, +4 CIDEr on NoCaps and nearly a +3\% accuracy on specific VQA question types. This underscores the efficacy of applying RAG approaches to VLMs, marking a stride toward more efficient and accessible multimodal learning.