Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning
This addresses the problem of automating piano accompaniment creation for musicians or producers, but it is incremental as it builds on existing cross-modal and representation learning approaches.
The paper tackles the audio-to-symbolic arrangement problem by automatically deriving piano scores from pop song audio, using a cross-modal representation-learning model that extracts chord and melodic information and learns texture from audio and corrupted scores, resulting in improved generation quality over baselines.
Could we automatically derive the score of a piano accompaniment based on the audio of a pop song? This is the audio-to-symbolic arrangement problem we tackle in this paper. A good arrangement model should not only consider the audio content but also have prior knowledge of piano composition (so that the generation "sounds like" the audio and meanwhile maintains musicality). To this end, we contribute a cross-modal representation-learning model, which 1) extracts chord and melodic information from the audio, and 2) learns texture representation from both audio and a corrupted ground truth arrangement. We further introduce a tailored training strategy that gradually shifts the source of texture information from corrupted score to audio. In the end, the score-based texture posterior is reduced to a standard normal distribution, and only audio is needed for inference. Experiments show that our model captures major audio information and outperforms baselines in generation quality.