The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
This work addresses the scalability and architectural simplicity of vision-language models for AI researchers, though it is incremental as it adapts existing mechanisms rather than introducing new components.
The paper tackles the complexity of multimodal large language models by introducing SAIL, a single transformer that integrates vision and language processing without a separate vision encoder, achieving performance comparable to modular models and matching ViT-22B in tasks like semantic segmentation.
This paper introduces SAIL, a single transformer unified multimodal large language model (MLLM) that integrates raw pixel encoding and language decoding within a singular architecture. Unlike existing modular MLLMs, which rely on a pre-trained vision transformer (ViT), SAIL eliminates the need for a separate vision encoder, presenting a more minimalist architecture design. Instead of introducing novel architectural components, SAIL adapts mix-attention mechanisms and multimodal positional encodings to better align with the distinct characteristics of visual and textual modalities. We systematically compare SAIL's properties-including scalability, cross-modal information flow patterns, and visual representation capabilities-with those of modular MLLMs. By scaling both training data and model size, SAIL achieves performance comparable to modular MLLMs. Notably, the removal of pretrained ViT components enhances SAIL's scalability and results in significantly different cross-modal information flow patterns. Moreover, SAIL demonstrates strong visual representation capabilities, achieving results on par with ViT-22B in vision tasks such as semantic segmentation. Code and models are available at https://github.com/bytedance/SAIL.