Speech Synthesis with Mixed Emotions
This research addresses the need for more nuanced emotional speech synthesis, enabling mixed emotions rather than single types, which is a novel contribution in the field.
The paper tackles the problem of synthesizing speech with mixed emotions at run-time, proposing a novel formulation to measure emotional differences and incorporating it into a sequence-to-sequence framework, with objective and subjective evaluations validating its effectiveness.
Emotional speech synthesis aims to synthesize human voices with various emotional effects. The current studies are mostly focused on imitating an averaged style belonging to a specific emotion type. In this paper, we seek to generate speech with a mixture of emotions at run-time. We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. During the training, the framework does not only explicitly characterize emotion styles, but also explores the ordinal nature of emotions by quantifying the differences with other emotions. At run-time, we control the model to produce the desired emotion mixture by manually defining an emotion attribute vector. The objective and subjective evaluations have validated the effectiveness of the proposed framework. To our best knowledge, this research is the first study on modelling, synthesizing, and evaluating mixed emotions in speech.