Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
This addresses the problem of computational and time-consuming training for multimodal models, offering an efficient paradigm for researchers and developers, though it appears incremental as it builds on existing MLLM advancements.
The paper tackles the challenge of building powerful Multimodal Large Language Models (MLLMs) efficiently by introducing Capybara-OMNI, which trains in a lightweight manner and supports text, image, video, and audio modalities, achieving competitive performance on various benchmarks for models of the same scale.
With the development of Multimodal Large Language Models (MLLMs), numerous outstanding accomplishments have emerged within the open-source community. Due to the complexity of creating and training multimodal data pairs, it is still a computational and time-consuming process to build powerful MLLMs. In this work, we introduce Capybara-OMNI, an MLLM that trains in a lightweight and efficient manner and supports understanding text, image, video, and audio modalities. We present in detail the framework design, the data construction, and the training recipe, to develop an MLLM step-by-step to obtain competitive performance. We also provide exclusive benchmarks utilized in our experiments to show how to properly verify understanding capabilities across different modalities. Results show that by following our guidance, we can efficiently build an MLLM that achieves competitive performance among models of the same scale on various multimodal benchmarks. Additionally, to enhance the multimodal instruction following and conversational capabilities of the model, we further discuss how to train the chat version upon an MLLM understanding model, which is more in line with user habits for tasks like real-time interaction with humans. We publicly disclose the Capybara-OMNI model, along with its chat-based version. The disclosure includes both the model weights, a portion of the training data, and the inference codes, which are made available on GitHub.