Applying the Information Bottleneck Principle to Prosodic Representation Learning
This work addresses controllable neural speech generation for applications like text-to-speech, but it is incremental as it builds on existing VQ-VAE methods.
The paper tackles the problem of learning prosodic representations from speech data by applying the information bottleneck principle, resulting in a model that can reconstruct speech and transfer prosody to different text with optimized performance.
This paper describes a novel design of a neural network-based speech generation model for learning prosodic representation.The problem of representation learning is formulated according to the information bottleneck (IB) principle. A modified VQ-VAE quantized layer is incorporated in the speech generation model to control the IB capacity and adjust the balance between reconstruction power and disentangle capability of the learned representation. The proposed model is able to learn word-level prosodic representations from speech data. With an optimized IB capacity, the learned representations not only are adequate to reconstruct the original speech but also can be used to transfer the prosody onto different textual content. Extensive results of the objective and subjective evaluation are presented to demonstrate the effect of IB capacity control, the effectiveness, and potential usage of the learned prosodic representation in controllable neural speech generation.