Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language
This addresses the problem of accurate pitch accent generation in Japanese TTS, which is crucial for natural speech synthesis in pitch accent languages, but is incremental as it adapts an existing model.
The study tackled the challenge of rendering correct pitch accents in Japanese end-to-end text-to-speech synthesis by investigating PnG BERT, a pretrained model, and found that it outperformed a baseline Tacotron model on accent correctness in listening tests.
End-to-end text-to-speech synthesis (TTS) can generate highly natural synthetic speech from raw text. However, rendering the correct pitch accents is still a challenging problem for end-to-end TTS. To tackle the challenge of rendering correct pitch accent in Japanese end-to-end TTS, we adopt PnG~BERT, a self-supervised pretrained model in the character and phoneme domain for TTS. We investigate the effects of features captured by PnG~BERT on Japanese TTS by modifying the fine-tuning condition to determine the conditions helpful inferring pitch accents. We manipulate content of PnG~BERT features from being text-oriented to speech-oriented by changing the number of fine-tuned layers during TTS. In addition, we teach PnG~BERT pitch accent information by fine-tuning with tone prediction as an additional downstream task. Our experimental results show that the features of PnG~BERT captured by pretraining contain information helpful inferring pitch accent, and PnG~BERT outperforms baseline Tacotron on accent correctness in a listening test.