TADA! Tuning Audio Diffusion Models through Activation Steering
This work addresses the challenge of interpretability and control in audio generation for researchers and practitioners, though it is incremental as it builds on existing activation steering methods.
The authors tackled the problem of understanding and controlling high-level musical concepts in audio diffusion models by identifying that specific semantic features are controlled by a shared subset of attention layers, and they demonstrated that steering activations in these layers allows precise modulation of elements like tempo or mood.
Audio diffusion models can synthesize high-fidelity music from text, yet their internal mechanisms for representing high-level concepts remain poorly understood. In this work, we use activation patching to demonstrate that distinct semantic musical concepts, such as the presence of specific instruments, vocals, or genre characteristics, are controlled by a small, shared subset of attention layers in state-of-the-art audio diffusion architectures. Next, we demonstrate that applying Contrastive Activation Addition and Sparse Autoencoders in these layers enables more precise control over the generated audio, indicating a direct benefit of the specialization phenomenon. By steering activations of the identified layers, we can alter specific musical elements with high precision, such as modulating tempo or changing a track's mood.