LLMBind: A Unified Modality-Task Integration Framework
This addresses user confusion and progress hindrance in multi-modal AI by providing a unified framework, though it appears incremental as it builds on existing MoE and LLM techniques.
The paper tackles the challenge of diverse input formats in multi-modal AI by introducing LLMBind, a framework that unifies multi-modal tasks using a Mixture-of-Experts LLM, achieving superior performance in user evaluations and enabling interactive visual generation with a 400k-instruction dataset.
In the multi-modal domain, the dependence of various models on specific input formats leads to user confusion and hinders progress. To address this challenge, we introduce \textbf{LLMBind}, a novel framework designed to unify a diverse array of multi-modal tasks. By harnessing a Mixture-of-Experts (MoE) Large Language Model (LLM), LLMBind processes multi-modal inputs and generates task-specific tokens, enabling the invocation of corresponding models to accomplish tasks. This unique approach empowers LLMBind to interpret inputs and generate outputs across various modalities, including image, text, video, and audio. Furthermore, we have constructed an interaction dataset comprising 400k instructions, which unlocks the ability of LLMBind for interactive visual generation and editing tasks. Extensive experimentation demonstrates that LLMBind achieves very superior performance across diverse tasks and outperforms existing models in user evaluations conducted in real-world scenarios. Moreover, the adaptability of LLMBind allows for seamless integration with the latest models and extension to new modality tasks, highlighting its potential to serve as a unified AI agent for modeling universal modalities.