Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation
This work addresses the need for efficient and accurate speech annotation in Japanese TTS, though it is incremental as it builds on existing ASR models and decoding techniques.
The paper tackles the problem of annotating phonemic and prosodic labels for Japanese TTS datasets by fine-tuning a pre-trained ASR model with transcript conditioning and dictionary-enhanced decoding, achieving objective improvements over previous methods and subjective naturalness comparable to manual annotations.
In this paper, we propose a method for annotating phonemic and prosodic labels on a given audio-transcript pair, aimed at constructing Japanese text-to-speech (TTS) datasets. Our approach involves fine-tuning a large-scale pre-trained automatic speech recognition (ASR) model, conditioned on ground truth transcripts, to simultaneously output phrase-level graphemes and annotation labels. To further correct errors in phonemic labeling, we employ a decoding strategy that utilizes dictionary prior knowledge. The objective evaluation results demonstrate that our proposed method outperforms previous approaches relying solely on text or audio. The subjective evaluation results indicate that the naturalness of speech synthesized by the TTS model, trained with labels annotated using our method, is comparable to that of a model trained with manual annotations.