Online Symbolic Music Alignment with Offline Reinforcement Learning
This addresses the challenge of real-time music synchronization for applications like score following, though it appears incremental as it builds on existing RL and alignment methods.
The paper tackles the problem of online symbolic music alignment by introducing a reinforcement learning-based technique that matches performed MIDI notes to score notes in real-time, achieving performance that outperforms a state-of-the-art offline reference model.
Symbolic Music Alignment is the process of matching performed MIDI notes to corresponding score notes. In this paper, we introduce a reinforcement learning (RL)-based online symbolic music alignment technique. The RL agent - an attention-based neural network - iteratively estimates the current score position from local score and performance contexts. For this symbolic alignment task, environment states can be sampled exhaustively and the reward is dense, rendering a formulation as a simplified offline RL problem straightforward. We evaluate the trained agent in three ways. First, in its capacity to identify correct score positions for sampled test contexts; second, as the core technique of a complete algorithm for symbolic online note-wise alignment; and finally, as a real-time symbolic score follower. We further investigate the pitch-based score and performance representations used as the agent's inputs. To this end, we develop a second model, a two-step Dynamic Time Warping (DTW)-based offline alignment algorithm leveraging the same input representation. The proposed model outperforms a state-of-the-art reference model of offline symbolic music alignment.