Semantic Codebooks as Effective Priors for Neural Speech Compression
This work addresses the problem of inefficient compression for speech codecs, benefiting applications in audio processing and recognition, though it is incremental by building on existing neural codec methods.
The paper tackled inefficient speech compression by proposing SemDAC, a semantic-aware neural audio codec that uses semantic codebooks as priors, resulting in improved perceptual metrics and lower word error rates at bitrates as low as 0.95 kbps compared to 2.5 kbps for DAC.
Speech codecs are traditionally optimized for waveform fidelity, allocating bits to preserve acoustic detail even when much of it can be inferred from linguistic structure. This leads to inefficient compression and suboptimal performance on downstream recognition tasks. We propose SemDAC, a semantic-aware neural audio codec that leverages semantic codebooks as effective priors for speech compression. In SemDAC, the first quantizer in a residual vector quantization (RVQ) stack is distilled from HuBERT features to produce semantic tokens that capture phonetic content, while subsequent quantizers model residual acoustics. A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks. Despite its simplicity, this design proves highly effective: SemDAC outperforms DAC across perceptual metrics and achieves lower WER when running Whisper on reconstructed speech, all while operating at substantially lower bitrates (e.g., 0.95 kbps vs. 2.5 kbps for DAC). These results demonstrate that semantic codebooks provide an effective inductive bias for neural speech compression, producing compact yet recognition-friendly representations.