Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models
This work addresses the problem of computational inefficiency and limited control in AI music generation for music producers, offering an incremental improvement by adapting models to real-world production needs.
The paper tackles the challenge of generating high-quality musical accompaniments that fit existing workflows by introducing Diff-A-Riff, a Latent Diffusion Model that produces 48kHz pseudo-stereo audio with reduced inference time and memory usage, as validated through objective metrics and listening tests.
Recent advancements in deep generative models present new opportunities for music production but also pose challenges, such as high computational demands and limited audio quality. Moreover, current systems frequently rely solely on text input and typically focus on producing complete musical pieces, which is incompatible with existing workflows in music production. To address these issues, we introduce "Diff-A-Riff," a Latent Diffusion Model designed to generate high-quality instrumental accompaniments adaptable to any musical context. This model offers control through either audio references, text prompts, or both, and produces 48kHz pseudo-stereo audio while significantly reducing inference time and memory usage. We demonstrate the model's capabilities through objective metrics and subjective listening tests, with extensive examples available on the accompanying website: sonycslparis.github.io/diffariff-companion/