WhispSynth: Scaling Multilingual Whisper Corpus through Real Data Curation and A Novel Pitch-free Generative Framework
This addresses the data scarcity problem for text-to-whisper research, particularly in multilingual contexts, though it appears incremental as it builds on existing TTS and DDSP methods.
The paper tackles the difficulty of collecting high-fidelity whispered speech data by introducing WhispSynth, a large-scale multilingual corpus of 118 hours from 479 speakers, generated through a novel pitch-free framework that preserves vocal timbre and linguistic content, with experiments showing significantly higher quality than existing corpora and a tuned model achieving speech naturalness on par with ground-truth samples.
Whisper generation is constrained by the difficulty of data collection. Because whispered speech has low acoustic amplitude, high-fidelity recording is challenging. In this paper, we introduce WhispSynth, a large-scale multilingual corpus constructed via a novel high-fidelity generative framework. Specifically, we propose a pipeline integrating Differentiable Digital Signal Processing (DDSP)-based pitch-free method with Text-to-Speech (TTS) models. This framework refines a comprehensive collection of resources, including our newly constructed WhispNJU dataset, into 118 hours of high-fidelity whispered speech from 479 speakers. Unlike standard synthetic or noisy real data, our data engine faithfully preserves source vocal timbre and linguistic content while ensuring acoustic consistency, providing a robust foundation for text-to-whisper research. Experimental results demonstrate that WhispSynth exhibits significantly higher quality than existing corpora. Moreover, our CosyWhisper, tuned with WhispSynth, achieves speech naturalness on par with ground-truth samples. The official implementation and related resources are available at https://github.com/tan90xx/cosywhisper.