LL-SDR: Low-Latency Speech enhancement through Discrete Representations
This work addresses speech enhancement for applications requiring real-time processing, but it is incremental as it builds on existing token-based methods.
The paper tackles speech enhancement by introducing LL-SDR, a token-based framework that uses discretization to separate speech and noise, achieving performance matching autoregressive token-based approaches while enabling low-latency enhancement in noisy environments.
Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.