SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
This work addresses a fundamental limitation in MLLMs for researchers and developers, offering an incremental improvement in alignment strategies.
The paper tackles the problem of suboptimal modality alignment in Multimodal Large Language Models (MLLMs), which limits their ability to interpret visual features, especially in smaller models, and proposes Supervised Embedding Alignment (SEA) to improve token-level visual-textual integration, resulting in an average performance gain of 7.61% for Gemma-2B.
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual inputs, yet modality alignment remains one of the most challenging aspects. Current MLLMs typically rely on simple adapter architectures and pretraining approaches to bridge vision encoders with large language models (LLM), guided by image-level supervision. We identify this paradigm often leads to suboptimal alignment between modalities, significantly constraining the LLM's ability to properly interpret and reason with visual features particularly for smaller language models. This limitation degrades overall performance-particularly for smaller language models where capacity constraints are more pronounced and adaptation capabilities are limited. To address this fundamental limitation, we propose Supervised Embedding Alignment (SEA), a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. SEA introduces minimal computational overhead while preserving language capabilities and substantially improving cross-modal understanding. Our comprehensive analyses reveal critical insights into the adapter's role in multimodal integration, and extensive experiments demonstrate that SEA consistently improves performance across various model sizes, with smaller models benefiting the most (average performance gain of 7.61% for Gemma-2B). This work establishes a foundation for developing more effective alignment strategies for future multimodal systems.