Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation
This work addresses the challenge of stem compatibility estimation for music production and analysis, offering a novel method that is incremental in applying self-supervised learning to this specific domain.
The paper tackles the problem of automatically determining which single-instrument audio recordings blend well with a given musical mix, and presents Stem-JEPA, a novel joint-embedding predictive architecture that achieves strong performance on stem retrieval tasks, as demonstrated on the MUSDB18 dataset and through subjective user studies.
This paper explores the automated process of determining stem compatibility by identifying audio recordings of single instruments that blend well with a given musical context. To tackle this challenge, we present Stem-JEPA, a novel Joint-Embedding Predictive Architecture (JEPA) trained on a multi-track dataset using a self-supervised learning approach. Our model comprises two networks: an encoder and a predictor, which are jointly trained to predict the embeddings of compatible stems from the embeddings of a given context, typically a mix of several instruments. Training a model in this manner allows its use in estimating stem compatibility - retrieving, aligning, or generating a stem to match a given mix - or for downstream tasks such as genre or key estimation, as the training paradigm requires the model to learn information related to timbre, harmony, and rhythm. We evaluate our model's performance on a retrieval task on the MUSDB18 dataset, testing its ability to find the missing stem from a mix and through a subjective user study. We also show that the learned embeddings capture temporal alignment information and, finally, evaluate the representations learned by our model on several downstream tasks, highlighting that they effectively capture meaningful musical features.