LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement
This addresses the problem of improving generalization across diverse speech enhancement tasks for researchers and practitioners, representing a novel method rather than an incremental improvement.
The paper tackles the problem of acoustic inconsistency and limited generalization in language model-based speech enhancement by introducing LLaSE-G1, which uses continuous representations and dual-channel inputs/outputs to unify multiple tasks, achieving state-of-the-art performance and emerging capabilities for unseen tasks.
Recent advancements in language models (LMs) have demonstrated strong capabilities in semantic understanding and contextual modeling, which have flourished in generative speech enhancement (SE). However, many LM-based SE approaches primarily focus on semantic information, often neglecting the critical role of acoustic information, which leads to acoustic inconsistency after enhancement and limited generalization across diverse SE tasks. In this paper, we introduce LLaSE-G1, a LLaMA-based language model that incentivizes generalization capabilities for speech enhancement. LLaSE-G1 offers the following key contributions: First, to mitigate acoustic inconsistency, LLaSE-G1 employs continuous representations from WavLM as input and predicts speech tokens from X-Codec2, maximizing acoustic preservation. Second, to promote generalization capability, LLaSE-G1 introduces dual-channel inputs and outputs, unifying multiple SE tasks without requiring task-specific IDs. Third, LLaSE-G1 outperforms prior task-specific discriminative and generative SE models, demonstrating scaling effects at test time and emerging capabilities for unseen SE tasks. Additionally, we release our code and models to support further research in this area.