Why do LLaVA Vision-Language Models Reply to Images in English?
This addresses a problem for researchers and engineers developing inclusive VLMs for non-English contexts, though it is incremental as it focuses on diagnosing and mitigating an existing bias.
The paper investigates a multilingual bias in LLaVA-style vision-language models, where including an image in a query increases the likelihood of English responses regardless of the query language, and finds that switching to a bilingual language backbone significantly reduces this error.
We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models' internal representations of image and text inputs. Both approaches indicate that the issue stems in the language modelling component of the LLaVA model. Statistically, we find that switching the language backbone for a bilingual language model has the strongest effect on reducing this error. Mechanistically, we provide compelling evidence that visual inputs are not mapped to a similar space as text ones, and that intervening on intermediary attention layers can reduce this bias. Our findings provide important insights to researchers and engineers seeking to understand the crossover between multimodal and multilingual spaces, and contribute to the goal of developing capable and inclusive VLMs for non-English contexts.