Mechanisms of Object Localization in Vision-Language Models
For researchers and practitioners working on vision-language models, this work provides mechanistic insights into object localization, revealing narrow computational pathways that could inform model design and grounding objectives.
The paper investigates object localization mechanisms in VLMs (LLaVA-1.5 and InternVL-3.5), finding that localization relies on a containerization mechanism where object-aligned tokens define spatial extent, and only a small set of attention heads mediate the causal effect. This is the first layer- and head-level account of localization in VLMs.
Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.