Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
This research provides insights into the internal workings of VLMs for researchers and developers, specifically pinpointing the OCR routing bottleneck and its potential interference with other visual tasks.
This paper investigates where OCR information is integrated into the language processing stream within vision-language models (VLMs) across three architecture families. They found that DeepStack models (Qwen) show peak sensitivity to scene text at mid-depth (around 50%), while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%). The OCR signal is low-dimensional, with PC1 capturing 72.9% of variance, and its removal can improve counting performance by up to 6.9 percentage points in modular architectures.
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.