Self-supervised Context-aware Style Representation for Expressive Speech Synthesis
This work addresses the problem of reducing annotation costs and improving style representation for expressive speech synthesis, particularly in audiobook applications, though it is incremental as it builds on existing multi-style TTS methods.
The paper tackles the challenge of learning style representations for expressive speech synthesis without costly labeled data by proposing a self-supervised framework that uses plain text, an emotion lexicon, contrastive learning, and deep clustering. It achieves improved subjective evaluation results on in-domain and out-of-domain audiobook test sets compared to methods relying on human-annotated style tags, with more natural emotion transitions in long paragraphs.
Expressive speech synthesis, like audiobook synthesis, is still challenging for style representation learning and prediction. Deriving from reference audio or predicting style tags from text requires a huge amount of labeled data, which is costly to acquire and difficult to define and annotate accurately. In this paper, we propose a novel framework for learning style representation from abundant plain text in a self-supervised manner. It leverages an emotion lexicon and uses contrastive learning and deep clustering. We further integrate the style representation as a conditioned embedding in a multi-style Transformer TTS. Comparing with multi-style TTS by predicting style tags trained on the same dataset but with human annotations, our method achieves improved results according to subjective evaluations on both in-domain and out-of-domain test sets in audiobook speech. Moreover, with implicit context-aware style representation, the emotion transition of synthesized audio in a long paragraph appears more natural. The audio samples are available on the demo web.