Remixing Music with Visual Conditioning
This work addresses music production and editing tasks for users by enabling remixing with visual inputs, but it is incremental as it builds on existing audio-visual source separation models.
The paper tackles the problem of music remixing by developing a system that uses visual conditioning from user-selected images to separate instrument sources from audio-only content, achieving improved audio quality compared to a baseline separate-and-add method.
We propose a visually conditioned music remixing system by incorporating deep visual and audio models. The method is based on a state of the art audio-visual source separation model which performs music instrument source separation with video information. We modified the model to work with user-selected images instead of videos as visual input during inference to enable separation of audio-only content. Furthermore, we propose a remixing engine that generalizes the task of source separation into music remixing. The proposed method is able to achieve improved audio quality compared to remixing performed by the separate-and-add method with a state-of-the-art audio-visual source separation model.