LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation
This work addresses the need for efficient and flexible control mechanisms in musical audio generation, offering a modular solution that enables more efficient training and deployment, though it is incremental as it builds on existing ControlNet methods.
The paper tackles the problem of fine-grained, time-varying control in text-to-audio diffusion models for music generation, proposing a lightweight architecture that reduces parameter count and memory usage while matching ControlNet in audio quality and condition adherence.
Text-to-audio diffusion models produce high-quality and diverse music but many, if not most, of the SOTA models lack the fine-grained, time-varying controls essential for music production. ControlNet enables attaching external controls to a pre-trained generative model by cloning and fine-tuning its encoder on new conditionings. However, this approach incurs a large memory footprint and restricts users to a fixed set of controls. We propose a lightweight, modular architecture that considerably reduces parameter count while matching ControlNet in audio quality and condition adherence. Our method offers greater flexibility and significantly lower memory usage, enabling more efficient training and deployment of independent controls. We conduct extensive objective and subjective evaluations and provide numerous audio examples on the accompanying website at https://lightlatentcontrol.github.io