Optimal Transport Maps are Good Voice Converters
This work addresses voice conversion for speech processing applications, presenting an incremental improvement by adapting existing optimal transport methods to this domain.
The paper tackles voice conversion by applying optimal transport maps to different speech data representations, achieving strong Frechet Audio Distance results for mel-spectrograms and state-of-the-art performance in latent spaces with limited reference data.
Recently, neural network-based methods for computing optimal transport maps have been effectively applied to style transfer problems. However, the application of these methods to voice conversion is underexplored. In our paper, we fill this gap by investigating optimal transport as a framework for voice conversion. We present a variety of optimal transport algorithms designed for different data representations, such as mel-spectrograms and latent representation of self-supervised speech models. For the mel-spectogram data representation, we achieve strong results in terms of Frechet Audio Distance (FAD). This performance is consistent with our theoretical analysis, which suggests that our method provides an upper bound on the FAD between the target and generated distributions. Within the latent space of the WavLM encoder, we achived state-of-the-art results and outperformed existing methods even with limited reference speaker data.