DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding
This work addresses efficiency in speech generation for applications requiring high-quality audio, but it is incremental as it builds on existing tokenization and diffusion methods.
The paper tackles the problem of inefficient speech tokenization in non-streaming scenarios by proposing DiffSoundStream, which conditions a neural codec on semantic tokens and uses latent diffusion models to synthesize waveforms, achieving speech quality comparable to a standard model at half the token rate (50 tokens per second vs. 100 tokens per second).
Token-based language modeling is a prominent approach for speech generation, where tokens are obtained by quantizing features from self-supervised learning (SSL) models and extracting codes from neural speech codecs, generally referred to as semantic tokens and acoustic tokens. These tokens are often modeled autoregressively, with the inference speed being constrained by the token rate. In this work, we propose DiffSoundStream, a solution that improves the efficiency of speech tokenization in non-streaming scenarios through two techniques: (1) conditioning the neural codec on semantic tokens to minimize redundancy between semantic and acoustic tokens, and (2) leveraging latent diffusion models to synthesize high-quality waveforms from semantic and coarse-level acoustic tokens. Experiments show that at 50 tokens per second, DiffSoundStream achieves speech quality on par with a standard SoundStream model operating at twice the token rate. Additionally, we achieve step-size distillation using just four diffusion sampling steps with only a minor quality loss.