AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation
This work addresses the need for efficient, high-quality speech compression with disentangled representations, benefiting speech communication and analysis applications.
AugCodec proposes a low-bitrate neural speech codec that uses data augmentation to disentangle speech into semantic, speaker, and prosody tokens, achieving high reconstruction quality and disentanglement at 12.5Hz with three token streams, outperforming state-of-the-art methods on LibriSpeech test-clean.
We propose AugCodec, a low-bitrate disentangled neural speech codec that leverages data augmentation to decompose speech into three distinct components: semantic, speaker, and prosody tokens. Specifically, we employ tailored augmenta tion strategies to transform speech into distinct variants, each serving as input for extracting tokens that preserve the target attribute while suppressing others. This disentanglement strategy enables substantial reduction in token rate. Further more, we introduce an augmentation loss that aligns semantic encoder outputs between source and voice-converted speech, encouraging speaker-agnostic embeddings while mitigating the acoustic mismatch induced by voice conversion. Experiments on LibriSpeech test-clean demonstrate that AugCodec significantly outperforms state-of-the-art methods in both reconstruction quality and disentanglement, while operating at only 12.5Hz with three token streams.