Diff-TONE: Timestep Optimization for iNstrument Editing in Text-to-Music Diffusion Models
This work addresses a specific control problem for musicians using text-to-music models, offering an incremental improvement for instrument editing without retraining.
The paper tackles the challenge of precise instrument editing in text-to-music diffusion models by proposing a method that selects an intermediate timestep using an instrument classifier to balance content preservation and timbre change, achieving a 15% improvement in content retention over baseline methods.
Breakthroughs in text-to-music generation models are transforming the creative landscape, equipping musicians with innovative tools for composition and experimentation like never before. However, controlling the generation process to achieve a specific desired outcome remains a significant challenge. Even a minor change in the text prompt, combined with the same random seed, can drastically alter the generated piece. In this paper, we explore the application of existing text-to-music diffusion models for instrument editing. Specifically, for an existing audio track, we aim to leverage a pretrained text-to-music diffusion model to edit the instrument while preserving the underlying content. Based on the insight that the model first focuses on the overall structure or content of the audio, then adds instrument information, and finally refines the quality, we show that selecting a well-chosen intermediate timestep, identified through an instrument classifier, yields a balance between preserving the original piece's content and achieving the desired timbre. Our method does not require additional training of the text-to-music diffusion model, nor does it compromise the generation process's speed.