Accompanied Singing Voice Synthesis with Fully Text-controlled Melody
This work addresses the impracticality of melody inputs in text-to-song synthesis for users, offering minimal requirements and maximum control flexibility, though it is incremental as it builds on existing singing voice synthesis methods.
The paper tackles the problem of generating accompanied singing voices without requiring impractical melody inputs like music scores, by introducing MelodyLM, a model that synthesizes high-quality song pieces using only text and a reference voice, achieving superior performance in objective and subjective metrics.
Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. Current TTSong methods, inherited from singing voice synthesis (SVS), require melody-related information that can sometimes be impractical, such as music scores or MIDI sequences. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies, achieving minimal user requirements and maximum control flexibility. MelodyLM explicitly models MIDI as the intermediate melody-related feature and sequentially generates vocal tracks in a language model manner, conditioned on textual and vocal prompts. The accompaniment music is subsequently synthesized by a latent diffusion model with hybrid conditioning for temporal alignment. With minimal requirements, users only need to input lyrics and a reference voice to synthesize a song sample. For full control, just input textual prompts or even directly input MIDI. Experimental results indicate that MelodyLM achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://melodylm666.github.io.