CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models
This addresses a performance gap in large vision-language models for vision-centric tasks, offering an incremental improvement with broad applicability across diverse architectures.
The paper tackles the problem of large vision-language models underperforming on vision-centric tasks like image classification compared to their base vision encoders, and proposes CARPE, a model-agnostic framework that improves generalization across classification and vision-language benchmarks through adaptive weighting of visual and textual modalities.
Recent advancements in Large Vision-Language Models (LVLMs) have pushed them closer to becoming general-purpose assistants. Despite their strong performance, LVLMs still struggle with vision-centric tasks such as image classification, underperforming compared to their base vision encoders, which are often CLIP-based models. To address this limitation, we propose Context-Aware Image Representation Prioritization via Ensemble (CARPE), a novel, model-agnostic framework which introduces vision-integration layers and a context-aware ensemble strategy to identify when to prioritize image representations or rely on the reasoning capabilities of the language model. This design enhances the model's ability to adaptively weight visual and textual modalities and enables the model to capture various aspects of image representations, leading to consistent improvements in generalization across classification and vision-language benchmarks. Extensive experiments demonstrate that CARPE not only improves performance on image classification benchmarks but also enhances results across various vision-language benchmarks. Finally, CARPE is designed to be effectively integrated with most open-source LVLMs that consist of a vision encoder and a language model, ensuring its adaptability across diverse architectures.