On Prosody Modeling for ASR+TTS based Voice Conversion
This work addresses the issue of speaker mismatch in prosody transfer for voice conversion, which is important for improving speech naturalness and conversion similarity in applications like speech synthesis.
The paper tackled the problem of prosody modeling in ASR+TTS voice conversion, which was previously overlooked, by proposing a target text prediction (TTP) method to predict prosody from linguistic representations in a target-speaker-dependent manner, resulting in improved performance on the VCC2020 benchmark as shown in objective and subjective evaluations.
In voice conversion (VC), an approach showing promising results in the latest voice conversion challenge (VCC) 2020 is to first use an automatic speech recognition (ASR) model to transcribe the source speech into the underlying linguistic contents; these are then used as input by a text-to-speech (TTS) system to generate the converted speech. Such a paradigm, referred to as ASR+TTS, overlooks the modeling of prosody, which plays an important role in speech naturalness and conversion similarity. Although some researchers have considered transferring prosodic clues from the source speech, there arises a speaker mismatch during training and conversion. To address this issue, in this work, we propose to directly predict prosody from the linguistic representation in a target-speaker-dependent manner, referred to as target text prediction (TTP). We evaluate both methods on the VCC2020 benchmark and consider different linguistic representations. The results demonstrate the effectiveness of TTP in both objective and subjective evaluations.