KaraTuner: Towards end to end natural pitch correction for singing voice in karaoke
This work addresses the need for more natural and automated pitch correction in karaoke systems, representing an incremental improvement by integrating neural components into existing pipelines.
The authors tackled the problem of automatic pitch correction for singing voice in karaoke by developing KaraTuner, an end-to-end neural architecture that predicts pitch curves and resynthesizes audio, achieving higher preference than rule-based methods in A/B tests and significant advantages in timbre consistency and sound quality over other vocoders.
An automatic pitch correction system typically includes several stages, such as pitch extraction, deviation estimation, pitch shift processing, and cross-fade smoothing. However, designing these components with strategies often requires domain expertise and they are likely to fail on corner cases. In this paper, we present KaraTuner, an end-to-end neural architecture that predicts pitch curve and resynthesizes the singing voice directly from the tuned pitch and vocal spectrum extracted from the original recordings. Several vital technical points have been introduced in KaraTuner to ensure pitch accuracy, pitch naturalness, timbre consistency, and sound quality. A feed-forward Transformer is employed in the pitch predictor to capture longterm dependencies in the vocal spectrum and musical note. We also develop a pitch-controllable vocoder based on a novel source-filter block and the Fre-GAN architecture. KaraTuner obtains a higher preference than the rule-based pitch correction approach through A/B tests, and perceptual experiments show that the proposed vocoder achieves significant advantages in timbre consistency and sound quality compared with the parametric WORLD vocoder, phase vocoder and CLPC vocoder.