Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding
This work addresses the challenge of generating high-fidelity singing voices for applications in music production and entertainment, though it is incremental as it builds on existing PPG-based methods.
The paper tackled the problem of limited style and naturalness in non-parallel singing voice conversion by using an acoustic reference encoder with HuBERT features and a contrastive predictive coding module, resulting in significantly improved naturalness and similarity to the target singer.
Recently, phonetic posteriorgrams (PPGs) based methods have been quite popular in non-parallel singing voice conversion systems. However, due to the lack of acoustic information in PPGs, style and naturalness of the converted singing voices are still limited. To solve these problems, in this paper, we utilize an acoustic reference encoder to implicitly model singing characteristics. We experiment with different auxiliary features, including mel spectrograms, HuBERT, and the middle hidden feature (PPG-Mid) of pretrained automatic speech recognition (ASR) model, as the input of the reference encoder, and finally find the HuBERT feature is the best choice. In addition, we use contrastive predictive coding (CPC) module to further smooth the voices by predicting future observations in latent space. Experiments show that, compared with the baseline models, our proposed model can significantly improve the naturalness of converted singing voices and the similarity with the target singer. Moreover, our proposed model can also make the speakers with just speech data sing.