Adaptive Slimming for Scalable and Efficient Speech Enhancement
This work addresses the need for scalable and efficient speech enhancement for applications like hearing aids and real-time communication, representing an incremental improvement over static methods.
The paper tackles the problem of deploying speech enhancement systems on resource-constrained devices by introducing dynamic slimming to the DEMUCS architecture, enabling adaptive performance/efficiency trade-offs; it achieves the same or better speech quality as a static 25% utilization model while reducing MACs by 29% when using 10% capacity on average.
Speech enhancement (SE) enables robust speech recognition, real-time communication, hearing aids, and other applications where speech quality is crucial. However, deploying such systems on resource-constrained devices involves choosing a static trade-off between performance and computational efficiency. In this paper, we introduce dynamic slimming to DEMUCS, a popular SE architecture, making it scalable and input-adaptive. Slimming lets the model operate at different utilization factors (UF), each corresponding to a different performance/efficiency trade-off, effectively mimicking multiple model sizes without the extra storage costs. In addition, a router subnet, trained end-to-end with the backbone, determines the optimal UF for the current input. Thus, the system saves resources by adaptively selecting smaller UFs when additional complexity is unnecessary. We show that our solution is Pareto-optimal against individual UFs, confirming the benefits of dynamic routing. When training the proposed dynamically-slimmable model to use 10% of its capacity on average, we obtain the same or better speech quality as the equivalent static 25% utilization while reducing MACs by 29%.