Understanding Counting Mechanisms in Large Language and Vision-Language Models
It addresses the problem of understanding numerical reasoning in AI models for researchers in interpretability and AI safety, though it is incremental as it builds on existing mechanistic interpretability work.
This paper investigates how large language and vision-language models handle counting tasks, revealing that they use internal counter mechanisms with latent positional information that emerges progressively across layers and can be transferred between contexts, influenced by structural cues like separators in text.
This paper examines how large language models (LLMs) and large vision-language models (LVLMs) represent and compute numerical information in counting tasks. We use controlled experiments with repeated textual and visual items and analyze model behavior through causal mediation and activation patching. To this end, we design a specialized tool, CountScope, for mechanistic interpretability of numerical content. Results show that individual tokens or visual features encode latent positional count information that can be extracted and transferred across contexts. Layerwise analyses reveal a progressive emergence of numerical representations, with lower layers encoding small counts and higher layers representing larger ones. We identify an internal counter mechanism that updates with each item, stored mainly in the final token or region and transferable between contexts. In LVLMs, numerical information also appears in visual embeddings, shifting between background and foreground regions depending on spatial composition. Models rely on structural cues such as separators in text, which act as shortcuts for tracking item counts and influence the accuracy of numerical predictions. Overall, counting emerges as a structured, layerwise process in LLMs and follows the same general pattern in LVLMs, shaped by the properties of the vision encoder.