Baichuan-Omni-1.5 Technical Report
This work addresses the need for integrated multimodal AI systems, but it appears incremental as it builds on existing methods with optimizations in data, tokenization, and training.
The paper tackles the problem of building an omni-modal model with understanding and audio generation capabilities, achieving state-of-the-art results by outperforming models like GPT4o-mini and MiniCPM-o 2.6 in comprehensive omni-modal capabilities and matching Qwen2-VL-72B on multimodal medical benchmarks.
We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.