MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
This work addresses the need for efficient and accurate control in music generation for AI and creative applications, representing an incremental improvement with specific gains.
The paper tackles the problem of precise conditioning in text-to-music generation by proposing MuseControlLite, a lightweight mechanism that improves control accuracy from 56.6% to 61.1% for melody control while using 6.75 times fewer trainable parameters than state-of-the-art methods.
We propose MuseControlLite, a lightweight mechanism designed to fine-tune text-to-music generation models for precise conditioning using various time-varying musical attributes and reference audio signals. The key finding is that positional embeddings, which have been seldom used by text-to-music generation models in the conditioner for text conditions, are critical when the condition of interest is a function of time. Using melody control as an example, our experiments show that simply adding rotary positional embeddings to the decoupled cross-attention layers increases control accuracy from 56.6% to 61.1%, while requiring 6.75 times fewer trainable parameters than state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion Transformer model of Stable Audio Open. We evaluate various forms of musical attribute control, audio inpainting, and audio outpainting, demonstrating improved controllability over MusicGen-Large and Stable Audio Open ControlNet at a significantly lower fine-tuning cost, with only 85M trainble parameters. Source code, model checkpoints, and demo examples are available at: https://musecontrollite.github.io/web/.