Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
This work addresses the challenge of automated music synchronization for applications in music education, performance analysis, and interactive systems, presenting a novel approach but with incremental improvements in a specific domain.
The paper tackles the problem of score following, which involves tracking a musical performance in audio against a symbolic score, by formulating it as a multimodal Markov Decision Process and applying deep reinforcement learning algorithms like A2C. The result is an end-to-end agent that learns from scratch to listen, read scores from images, and follow along, showing superiority over previous methods for raw sheet music images.
Score following is the process of tracking a musical performance (audio) with respect to a known symbolic representation (a score). We start this paper by formulating score following as a multimodal Markov Decision Process, the mathematical foundation for sequential decision making. Given this formal definition, we address the score following task with state-of-the-art deep reinforcement learning (RL) algorithms such as synchronous advantage actor critic (A2C). In particular, we design multimodal RL agents that simultaneously learn to listen to music, read the scores from images of sheet music, and follow the audio along in the sheet, in an end-to-end fashion. All this behavior is learned entirely from scratch, based on a weak and potentially delayed reward signal that indicates to the agent how close it is to the correct position in the score. Besides discussing the theoretical advantages of this learning paradigm, we show in experiments that it is in fact superior compared to previously proposed methods for score following in raw sheet music images.