BRAVE: Broadening the visual encoding of vision-language models
This addresses visual shortcomings like 'blindness' and hallucination in VLMs, offering a more versatile solution for multimodal AI applications, though it is incremental as it builds on existing encoder methods.
The paper tackles the limited visual encoding capabilities of vision-language models (VLMs) by introducing BRAVE, a method that consolidates features from multiple frozen vision encoders, achieving state-of-the-art performance on captioning and VQA benchmarks with fewer trainable parameters and more compressed representations.
Vision-language models (VLMs) are typically composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders, e.g. "blindness" to certain image features, visual hallucination, etc. To address these issues, we study broadening the visual encoding capabilities of VLMs. We first comprehensively benchmark several vision encoders with different inductive biases for solving VLM tasks. We observe that there is no single encoding configuration that consistently achieves top performance across different tasks, and encoders with different biases can perform surprisingly similarly. Motivated by this, we introduce a method, named BRAVE, that consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen LM. BRAVE achieves state-of-the-art performance on a broad range of captioning and VQA benchmarks and significantly reduces the aforementioned issues of VLMs, while requiring a smaller number of trainable parameters than existing methods and having a more compressed representation. Our results highlight the potential of incorporating different visual biases for a more broad and contextualized visual understanding of VLMs.