SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description
This addresses the problem of limited high-quality annotated speech data for researchers in speech-language multi-modal learning, though it is incremental as it builds on existing annotation methods.
The authors tackled the challenge of creating a large-scale dataset for fine-grained expressive speech by developing an automatic annotation system that generates natural language descriptions for speech clips, resulting in the SpeechCraft dataset with approximately 2,000 hours of audio and over two million clips, which significantly boosts performance in speech-language tasks.
Speech-language multi-modal learning presents a significant challenge due to the fine nuanced information inherent in speech styles. Therefore, a large-scale dataset providing elaborate comprehension of speech style is urgently needed to facilitate insightful interplay between speech audio and natural language. However, constructing such datasets presents a major trade-off between large-scale data collection and high-quality annotation. To tackle this challenge, we propose an automatic speech annotation system for expressiveness interpretation that annotates in-the-wild speech clips with expressive and vivid human language descriptions. Initially, speech audios are processed by a series of expert classifiers and captioning models to capture diverse speech characteristics, followed by a fine-tuned LLaMA for customized annotation generation. Unlike previous tag/templet-based annotation frameworks with limited information and diversity, our system provides in-depth understandings of speech style through tailored natural language descriptions, thereby enabling accurate and voluminous data generation for large model training. With this system, we create SpeechCraft, a fine-grained bilingual expressive speech dataset. It is distinguished by highly descriptive natural language style prompts, containing approximately 2,000 hours of audio data and encompassing over two million speech clips. Extensive experiments demonstrate that the proposed dataset significantly boosts speech-language task performance in stylist speech synthesis and speech style understanding.