ModaVerse: Efficiently Transforming Modalities with LLMs
This work addresses the inefficiencies in training multi-modal models for researchers and practitioners, offering a more efficient approach, though it is incremental as it builds on existing LLM-as-agent methodologies.
The paper tackles the problem of multi-modal content transformation by introducing ModaVerse, a Multi-modal Large Language Model that uses a novel I/O alignment mechanism to simplify training, achieving comparable performance to state-of-the-art methods with significant reductions in data and computational costs.
Humans possess the capability to comprehend diverse modalities and seamlessly transfer information between them. In this work, we introduce ModaVerse, a Multi-modal Large Language Model (MLLM) capable of comprehending and transforming content across various modalities including images, videos, and audio. Predominant MLLM frameworks have largely relied on the alignment of latent spaces of textual and non-textual features. This alignment process, which synchronizes a language model trained on textual data with encoders and decoders trained on multi-modal data, often necessitates extensive training of several projection layers in multiple stages. Inspired by LLM-as-agent methodologies, we propose a novel Input/Output (I/O) alignment mechanism that operates directly at the level of natural language. It aligns the LLM's output with the input of generative models, avoiding the complexities associated with latent feature alignments, and simplifying the multiple training stages of existing MLLMs into a single, efficient process. This conceptual advancement leads to significant reductions in both data and computational costs. By conducting experiments on several benchmarks, we demonstrate that our approach attains comparable performance with the state of the art while achieving considerable efficiencies in data usage and training duration.