Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Model
This addresses the visual perception bottleneck in MLLMs for AI systems, offering a new design paradigm, though it is incremental as it builds on existing encoder fusion methods.
The paper tackles the problem of multimodal large language models (MLLMs) failing at basic visual tasks due to reliance on a single vision encoder, by introducing the Fusion to Enhance (FtZ) framework that composes a semantic anchor encoder with a perception-rich augmenting encoder. The result is significant performance improvements on benchmarks like TextVQA, POPE, MMMU, MME, and MM-Vet compared to single-encoder or existing fusion baselines.
Multimodal Large Language Models (MLLMs) have made significant progress in bridging visual perception with high-level textual reasoning. However, they face a fundamental contradiction: while excelling at complex semantic understanding, these models often fail at basic visual tasks that require precise detail perception. This deficiency primarily stems from the prevalent architectural reliance on a single vision encoder optimized for high-level semantic alignment, which inherently sacrifices the ability to capture fine-grained visual information. To address this issue, we introduce Fusion to Enhance (FtZ), a novel vision tower framework. FtZ moves beyond the single-encoder design by innovatively composing a semantically powerful anchor encoder with a perception-rich augmenting encoder via a lightweight Multi-Head Cross-Attention mechanism. Experimental results demonstrate that on several challenging benchmarks demanding fine-grained visual understanding, such as TextVQA, POPE, MMMU, MME and MM-Vet, our FtZ model significantly outperforms baselines that use only a single encoder or existing feature fusion methods. This work proves that composing heterogeneous expert encoders is an efficient and effective path to overcoming the visual perception bottleneck in current MLLMs, offering a new design paradigm for building next-generation AI systems with stronger perceptual capabilities.