Expressive TTS Driven by Natural Language Prompts Using Few Human Annotations
This work addresses the challenge of acquiring annotated data for controllable expressive TTS, offering a more adaptable solution for users seeking flexible style control with reduced human effort.
The paper tackles the problem of expressive text-to-speech synthesis requiring excessive style-annotated data by introducing FreeStyleTTS, which uses a large language model to retrieve styles from minimal annotations, resulting in synthetic speeches closely aligned with specified styles as demonstrated on a Mandarin storytelling corpus.
Expressive text-to-speech (TTS) aims to synthesize speeches with human-like tones, moods, or even artistic attributes. Recent advancements in expressive TTS empower users with the ability to directly control synthesis style through natural language prompts. However, these methods often require excessive training with a significant amount of style-annotated data, which can be challenging to acquire. Moreover, they may have limited adaptability due to fixed style annotations. In this work, we present FreeStyleTTS (FS-TTS), a controllable expressive TTS model with minimal human annotations. Our approach utilizes a large language model (LLM) to transform expressive TTS into a style retrieval task. The LLM selects the best-matching style references from annotated utterances based on external style prompts, which can be raw input text or natural language style descriptions. The selected reference guides the TTS pipeline to synthesize speeches with the intended style. This innovative approach provides flexible, versatile, and precise style control with minimal human workload. Experiments on a Mandarin storytelling corpus demonstrate FS-TTS's proficiency in leveraging LLM's semantic inference ability to retrieve desired styles from either input text or user-defined descriptions. This results in synthetic speeches that are closely aligned with the specified styles.