Ranking the Impact of Contextual Specialization in Neural Speech Enhancement
For resource-constrained applications like hearing aids, this work shows that small, contextually specialized models can achieve high performance, enabling on-the-fly adaptation.
This paper investigates how contextual specialization (e.g., to speaker identity, noise type, language) affects neural speech enhancement performance. It finds that specializing to a speaker's identity yields the largest gains, and a small specialized model can match a generalist model ten times its size.
We systematically investigate neural speech enhancement systems, ranging from very small ($\sim$10\,k parameters) to medium-large ($\sim$2-5\,M parameters), which specialize to acoustic conditions using contextual information such as speaker identity, noise type, speaker gender, spoken language, and SNR. By fine-tuning generalist models on specific data subsets, we find that specializing to a speaker's identity consistently yields the largest gains in estimated speech intelligibility and quality. In contrast, specializing to SNR, noise type, or gender offers only marginal benefits. Crucially, we show that a small model specialized to both a specific speaker and a specific noise type can match or exceed the performance of a generalist model ten times its size. Further, cross-lingual tests reveal that models specialized to a target language outperform multilingual generalists, suggesting that language is a salient feature for specialization. These findings highlight the potential of small, adaptive models for resource-constrained applications like hearing aids, which specialize on-the-fly to contextual information.