Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
This work addresses the challenging problem of unsupervised multi-instrumental music generation from video for musicians and creative AI applications, offering a novel approach to disentangle musical components and instrumental features.
The paper introduces Multi-instrumentalistNet (MI Net), a system that generates multi-instrumental music from musician body movements in an unsupervised setting. It successfully disentangles musical and instrumental features into distinct latent clusters, enabling the generation of music with reasonable audio quality that is easily associated with the corresponding instrument and consistent with the audio content across two datasets containing videos of 13 instruments.
We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting. Learning to generate multi-instrumental music from videos without labeling the instruments is a challenging problem. To achieve the transformation, we built a pipeline named 'Multi-instrumentalistNet' (MI Net). At its base, the pipeline learns a discrete latent representation of various instruments music from log-spectrogram using a Vector Quantized Variational Autoencoder (VQ-VAE) with multi-band residual blocks. The pipeline is then trained along with an autoregressive prior conditioned on the musician's body keypoints movements encoded by a recurrent neural network. Joint training of the prior with the body movements encoder succeeds in the disentanglement of the music into latent features indicating the musical components and the instrumental features. The latent space results in distributions that are clustered into distinct instruments from which new music can be generated. Furthermore, the VQ-VAE architecture supports detailed music generation with additional conditioning. We show that a Midi can further condition the latent space such that the pipeline will generate the exact content of the music being played by the instrument in the video. We evaluate MI Net on two datasets containing videos of 13 instruments and obtain generated music of reasonable audio quality, easily associated with the corresponding instrument, and consistent with the music audio content.