LLMs Can Compensate for Deficiencies in Visual Representations
This addresses the challenge of improving multimodal AI systems for researchers and developers by revealing a dynamic division of labor, though it is incremental as it builds on existing CLIP-based models.
The study tackled the problem of CLIP-based vision encoders' limitations in vision-language models by investigating whether the language backbone compensates for weak visual features, finding that the language decoder can largely recover performance in scenarios of reduced visual contextualization.
Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We investigate the hypothesis that the strong language backbone in VLMs compensates for possibly weak visual features by contextualizing or enriching them. Using three CLIP-based VLMs, we perform controlled self-attention ablations on a carefully designed probing task. Our findings show that despite known limitations, CLIP visual representations offer ready-to-read semantic information to the language decoder. However, in scenarios of reduced contextualization in the visual representations, the language decoder can largely compensate for the deficiency and recover performance. This suggests a dynamic division of labor in VLMs and motivates future architectures that offload more visual processing to the language decoder.