Interactive Text-to-Speech System via Joint Style Analysis
This addresses the issue of unnatural interactions in TTS for users, but it is incremental as it builds on existing TTS technologies with a focus on style adaptation.
The paper tackled the problem of low behavior naturalness in text-to-speech (TTS) systems by proposing a style-embedded TTS that generates responses matching the style of speech queries, using a semi-supervised approach to handle limited labeled data and disjoint datasets, with results showing user preference for styled responses.
While modern TTS technologies have made significant advancements in audio quality, there is still a lack of behavior naturalness compared to conversing with people. We propose a style-embedded TTS system that generates styled responses based on the speech query style. To achieve this, the system includes a style extraction model that extracts a style embedding from the speech query, which is then used by the TTS to produce a matching response. We faced two main challenges: 1) only a small portion of the TTS training dataset has style labels, which is needed to train a multi-style TTS that respects different style embeddings during inference. 2) The TTS system and the style extraction model have disjoint training datasets. We need consistent style labels across these two datasets so that the TTS can learn to respect the labels produced by the style extraction model during inference. To solve these, we adopted a semi-supervised approach that uses the style extraction model to create style labels for the TTS dataset and applied transfer learning to learn the style embedding jointly. Our experiment results show user preference for the styled TTS responses and demonstrate the style-embedded TTS system's capability of mimicking the speech query style.