Direct Noisy Speech Modeling for Noisy-to-Noisy Voice Conversion
This addresses noisy-to-noisy voice conversion for applications like movies or music where background sounds are informative, representing an incremental improvement over prior work.
The paper tackles the problem of speaker identity conversion in noisy speech while preserving background sounds, proposing an improved voice conversion module that directly models noisy waveforms to reduce distortion from denoising. Experimental results show it significantly outperforms the previous framework, achieving acceptable naturalness scores and comparable similarity to an upper bound.
Beyond the conventional voice conversion (VC) where the speaker information is converted without altering the linguistic content, the background sounds are informative and need to be retained in some real-world scenarios, such as VC in movie/video and VC in music where the voice is entangled with background sounds. As a new VC framework, we have developed a noisy-to-noisy (N2N) VC framework to convert the speaker's identity while preserving the background sounds. Although our framework consisting of a denoising module and a VC module well handles the background sounds, the VC module is sensitive to the distortion caused by the denoising module. To address this distortion issue, in this paper we propose the improved VC module to directly model the noisy speech waveform while controlling the background sounds. The experimental results have demonstrated that our improved framework significantly outperforms the previous one and achieves an acceptable score in terms of naturalness, while reaching comparable similarity performance to the upper bound of our framework.