Optimizing Vision-Language Interactions Through Decoder-Only Models
This addresses scalability and modality alignment problems for researchers and practitioners in multimodal AI, representing a novel architectural shift rather than an incremental improvement.
The paper tackled the inefficiency and alignment challenges in Vision-Language Models by proposing MUDAIF, a decoder-only model that integrates visual and textual inputs without a separate visual encoder, achieving state-of-the-art performance on benchmarks like VQA and image captioning with training on 45M image-text pairs.
Vision-Language Models (VLMs) have emerged as key enablers for multimodal tasks, but their reliance on separate visual encoders introduces challenges in efficiency, scalability, and modality alignment. To address these limitations, we propose MUDAIF (Multimodal Unified Decoder with Adaptive Input Fusion), a decoder-only vision-language model that seamlessly integrates visual and textual inputs through a novel Vision-Token Adapter (VTA) and adaptive co-attention mechanism. By eliminating the need for a visual encoder, MUDAIF achieves enhanced efficiency, flexibility, and cross-modal understanding. Trained on a large-scale dataset of 45M image-text pairs, MUDAIF consistently outperforms state-of-the-art methods across multiple benchmarks, including VQA, image captioning, and multimodal reasoning tasks. Extensive analyses and human evaluations demonstrate MUDAIF's robustness, generalization capabilities, and practical usability, establishing it as a new standard in encoder-free vision-language models.