Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
This work addresses a key bottleneck in MLLMs for improving multimodal AI applications, though it is incremental as it builds on existing paradigms like JEPA.
The paper tackles the limited visual reasoning in Multimodal Large Language Models (MLLMs) by introducing JARVIS, a self-supervised framework that enhances visual understanding without relying on textual descriptions, resulting in consistent performance improvements on vision-centric benchmarks across different LLM families.
Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in connecting vision and language, yet their proficiency in fundamental visual reasoning tasks remains limited. This limitation can be attributed to the fact that MLLMs learn visual understanding primarily from textual descriptions, which constitute a subjective and inherently incomplete supervisory signal. Furthermore, the modest scale of multimodal instruction tuning compared to massive text-only pre-training leads MLLMs to overfit language priors while overlooking visual details. To address these issues, we introduce JARVIS, a JEPA-inspired framework for self-supervised visual enhancement in MLLMs. Specifically, we integrate the I-JEPA learning paradigm into the standard vision-language alignment pipeline of MLLMs training. Our approach leverages frozen vision foundation models as context and target encoders, while training the predictor, implemented as the early layers of an LLM, to learn structural and semantic regularities from images without relying exclusively on language supervision. Extensive experiments on standard MLLM benchmarks show that JARVIS consistently improves performance on vision-centric benchmarks across different LLM families, without degrading multimodal reasoning abilities. Our source code is publicly available at: https://github.com/aimagelab/JARVIS.