Unified Multimodal Understanding via Byte-Pair Visual Encoding
This work addresses the fundamental problem of multimodal alignment for advancing vision-language AI systems, representing an incremental improvement with novel method elements.
The paper tackles the challenge of aligning different modalities in multimodal large language models by introducing a framework that applies byte-pair encoding to visual tokens, resulting in improved performance across diverse vision-language tasks.
Multimodal large language models (MLLMs) have made significant progress in vision-language understanding, yet effectively aligning different modalities remains a fundamental challenge. We present a framework that unifies multimodal understanding by applying byte-pair encoding to visual tokens. Unlike conventional approaches that rely on modality-specific encoders, our method directly incorporates structural information into visual tokens, mirroring successful tokenization strategies in text-only language models. We introduce a priority-guided encoding scheme that considers both frequency and spatial consistency, coupled with a multi-stage training procedure based on curriculum-driven data composition. These enhancements enable the transformer model to better capture cross-modal relationships and reason with visual information. Comprehensive experiments demonstrate improved performance across diverse vision-language tasks. By bridging the gap between visual and textual representations, our approach contributes to the advancement of more capable and efficient multimodal foundation models.