Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information
This addresses the problem of inefficient quantization in speech processing for applications like text-to-speech, offering a method to retain critical non-linguistic features without increasing bitrates.
The paper tackles the loss of prosodic and paralinguistic information (e.g., emotion, prominence) in quantization for SSL speech models like HuBERT, which improves compression but discards such details. The proposed Segmentation-Variant Codebooks (SVCs) quantize speech at distinct linguistic units, resulting in significantly better preservation of this information across probing tasks and improved style realization in resynthesis experiments.
Quantization in SSL speech models (e.g., HuBERT) improves compression and performance in tasks like language modeling, resynthesis, and text-to-speech but often discards prosodic and paralinguistic information (e.g., emotion, prominence). While increasing codebook size mitigates some loss, it inefficiently raises bitrates. We propose Segmentation-Variant Codebooks (SVCs), which quantize speech at distinct linguistic units (frame, phone, word, utterance), factorizing it into multiple streams of segment-specific discrete features. Our results show that SVCs are significantly more effective at preserving prosodic and paralinguistic information across probing tasks. Additionally, we find that pooling before rather than after discretization better retains segment-level information. Resynthesis experiments further confirm improved style realization and slightly improved quality while preserving intelligibility.