Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight
This addresses the computational bottleneck in training multimodal AI models, offering a practical speed-up for researchers and developers, though it is an incremental improvement over existing methods.
The paper tackles the slow pre-training of multimodal LLMs by introducing Chain-of-Sight, a vision-language bridge module that reduces visual tokens during pre-training, cutting training time by ~73% while maintaining or improving performance on benchmarks.
This paper introduces Chain-of-Sight, a vision-language bridge module that accelerates the pre-training of Multimodal Large Language Models (MLLMs). Our approach employs a sequence of visual resamplers that capture visual details at various spacial scales. This architecture not only leverages global and local visual contexts effectively, but also facilitates the flexible extension of visual tokens through a compound token scaling strategy, allowing up to a 16x increase in the token count post pre-training. Consequently, Chain-of-Sight requires significantly fewer visual tokens in the pre-training phase compared to the fine-tuning phase. This intentional reduction of visual tokens during pre-training notably accelerates the pre-training process, cutting down the wall-clock training time by ~73%. Empirical results on a series of vision-language benchmarks reveal that the pre-train acceleration through Chain-of-Sight is achieved without sacrificing performance, matching or surpassing the standard pipeline of utilizing all visual tokens throughout the entire training process. Further scaling up the number of visual tokens for pre-training leads to stronger performances, competitive to existing approaches in a series of benchmarks.