Detecting Music Performance Errors with Transformers
This addresses the challenge for beginner musicians in identifying specific performance errors, such as incorrect notes or rhythms, with a novel method that overcomes limitations in alignment and data scarcity.
The paper tackles the problem of detecting errors in music performances by proposing a transformer model that processes audio to output annotated scores and a data generation technique for synthetic error datasets, achieving a 64.1% average Error Detection F1 score and improving upon prior work by 40 percentage points across 14 instruments.
Beginner musicians often struggle to identify specific errors in their performances, such as playing incorrect notes or rhythms. There are two limitations in existing tools for music error detection: (1) Existing approaches rely on automatic alignment; therefore, they are prone to errors caused by small deviations between alignment targets.; (2) There is a lack of sufficient data to train music error detection models, resulting in over-reliance on heuristics. To address (1), we propose a novel transformer model, Polytune, that takes audio inputs and outputs annotated music scores. This model can be trained end-to-end to implicitly align and compare performance audio with music scores through latent space representations. To address (2), we present a novel data generation technique capable of creating large-scale synthetic music error datasets. Our approach achieves a 64.1% average Error Detection F1 score, improving upon prior work by 40 percentage points across 14 instruments. Additionally, compared with existing transcription methods repurposed for music error detection, our model can handle multiple instruments. Our source code and datasets are available at https://github.com/ben2002chou/Polytune.