DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
This addresses the challenge of building efficient and high-performing speech-text multimodal models for seamless human-computer interaction, representing an incremental improvement over existing methods.
The paper tackles the problem of catastrophic forgetting and performance degradation in native multimodal large language models (MLLMs) due to insufficient paired speech-text data, proposing DeepTalk, an adaptive modality expert learning framework based on Mixture of Experts (MoE) that reduces performance drop to 5.5% compared to over 20% in existing native MLLMs while maintaining end-to-end dialogue latency under 0.5 seconds.
Native multimodal large language models (MLLMs) restructure a single large language model (LLM) into a spoken language model (SLM) capable of both speech and text generation. Compared to modular and aligned MLLMs, native MLLMs preserve richer paralinguistic features such as emotion and prosody, and generate speech responses directly within the backbone LLM rather than using a separate speech decoder. This integration also results in lower response latency and smoother interaction. However, native MLLMs suffer from catastrophic forgetting and performance degradation because the available paired speech-text data is insufficient to support the pretraining of MLLMs compared to the vast amount of text data required to pretrain text LLMs. To address this issue, we propose DeepTalk, a framework for adaptive modality expert learning based on a Mixture of Experts (MoE) architecture. DeepTalk first adaptively distinguishes modality experts according to their modality load within the LLM. Each modality expert then undergoes specialized single-modality training, followed by joint multimodal collaborative training. As a result, DeepTalk incurs only a 5.5% performance drop compared to the original LLM, which is significantly lower than the average performance drop of over 20% typically seen in native MLLMs (such as GLM-4-Voice), and is on par with modular MLLMs. Meanwhile, the end-to-end dialogue latency remains within 0.5 seconds, ensuring a seamless and intelligent speech interaction experience. Code and models are released at https://github.com/talkking/DeepTalk.