Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
For researchers building multimodal LLMs, this work provides a theoretically grounded and empirically validated alternative to modular architectures, though it is demonstrated only on a specific biological domain.
The paper identifies the geometric modality gap as a bottleneck in multimodal LLMs and proposes One Tokenizer, a native architecture that maps all modalities into a shared token space, achieving zero-gap integration. On a DNA-text testbed, it consistently outperforms encoder-based modular counterparts.
A central challenge in developing Multimodal Large Language Models (MLLMs) is effectively integrating heterogeneous inputs into a cohesive reasoning engine. Current paradigms predominantly rely on modular architectures that introduce modality-specific encoders and cross-modal fusion mechanisms. However, these designs are fundamentally bottlenecked by a geometric modality gap, forcing the LLM to expend significant computational capacity on geometric reconciliation rather than deep cross-modal reasoning. In this work, we formally characterize this modality gap and theoretically demonstrate that native architectures, specifically those employing a unified vocabulary, intrinsically maintain a zero-gap state across all hidden layers. Guided by these theoretical findings, we propose \textit{One Tokenizer}, a native architecture that maps all modalities directly into a shared token space. We empirically validate this framework on a DNA--text multimodal testbed. Our extensive evaluations reveal that by achieving seamless integration within the LLM's native latent space, One Tokenizer consistently outperforms encoder-based modular counterparts, providing a fundamentally superior framework for deep biological reasoning.