What do vision-language models see in the context? Investigating multimodal in-context learning
This work addresses the underexplored effectiveness of ICL in VLMs, providing insights for researchers to enhance multimodal learning, though it is incremental as it builds on existing ICL research.
The study systematically investigates in-context learning (ICL) in vision-language models (VLMs) across seven models and three benchmarks, finding that training on interleaved data improves performance but does not ensure effective multimodal integration, while instruction tuning reduces reliance on demonstrations, highlighting limitations in leveraging visual information.
In-context learning (ICL) enables Large Language Models (LLMs) to learn tasks from demonstration examples without parameter updates. Although it has been extensively studied in LLMs, its effectiveness in Vision-Language Models (VLMs) remains underexplored. In this work, we present a systematic study of ICL in VLMs, evaluating seven models spanning four architectures on three image captioning benchmarks. We analyze how prompt design, architectural choices, and training strategies influence multimodal ICL. To our knowledge, we are the first to analyze how attention patterns in VLMs vary with an increasing number of in-context demonstrations. Our results reveal that training on imag-text interleaved data enhances ICL performance but does not imply effective integration of visual and textual information from demonstration examples. In contrast, instruction tuning improves instruction-following but can reduce reliance on in-context demonstrations, suggesting a trade-off between instruction alignment and in-context adaptation. Attention analyses further show that current VLMs primarily focus on textual cues and fail to leverage visual information, suggesting a limited capacity for multimodal integration. These findings highlight key limitations in the ICL abilities of current VLMs and provide insights for enhancing their ability to learn from multimodal in-context examples.