Continuous descriptor-based control for deep audio synthesis
This addresses the problem of limited usability for musicians by providing incremental improvements in controllability for creative workflows.
The paper tackles the lack of expressive control in deep audio synthesis for musicians by introducing a lightweight generative model that enables continuous descriptor-based control, akin to synthesizer knobs, and demonstrates performance on diverse sounds like instrumental, percussive, and speech recordings.
Despite significant advances in deep models for music generation, the use of these techniques remains restricted to expert users. Before being democratized among musicians, generative models must first provide expressive control over the generation, as this conditions the integration of deep generative models in creative workflows. In this paper, we tackle this issue by introducing a deep generative audio model providing expressive and continuous descriptor-based control, while remaining lightweight enough to be embedded in a hardware synthesizer. We enforce the controllability of real-time generation by explicitly removing salient musical features in the latent space using an adversarial confusion criterion. User-specified features are then reintroduced as additional conditioning information, allowing for continuous control of the generation, akin to a synthesizer knob. We assess the performance of our method on a wide variety of sounds including instrumental, percussive and speech recordings while providing both timbre and attributes transfer, allowing new ways of generating sounds.