Seeing to Generalize: How Visual Data Corrects Binding Shortcuts
This addresses the issue of poor generalization in language models for researchers and practitioners, but it is incremental as it builds on existing VLM and LLM frameworks.
The paper tackles the problem of Vision Language Models (VLMs) outperforming their underlying Large Language Models (LLMs) on text-only tasks, particularly in long-context information retrieval, by showing that training on image-tokenized data nearly doubles out-of-distribution text-only performance from a baseline that fails to generalize.
Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out of distribution, while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model's internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes, visual encoders, and initializations, and show that analogous shifts occur during pretrained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.