Quantifying the Corpus Bias Problem in Automatic Music Transcription Systems
This addresses the problem of overfitting in AMT systems for researchers and practitioners, but it is incremental as it builds on prior work on distribution shifts.
The paper tackled the corpus bias problem in automatic music transcription systems by evaluating state-of-the-art systems on new test sets with musical distribution shifts, revealing a stark performance gap that highlights generalization issues.
Automatic Music Transcription (AMT) is the task of recognizing notes in audio recordings of music. The State-of-the-Art (SotA) benchmarks have been dominated by deep learning systems. Due to the scarcity of high quality data, they are usually trained and evaluated exclusively or predominantly on classical piano music. Unfortunately, that hinders our ability to understand how they generalize to other music. Previous works have revealed several aspects of memorization and overfitting in these systems. We identify two primary sources of distribution shift: the music, and the sound. Complementing recent results on the sound axis (i.e. acoustics, timbre), we investigate the musical one (i.e. note combinations, dynamics, genre). We evaluate the performance of several SotA AMT systems on two new experimental test sets which we carefully construct to emulate different levels of musical distribution shift. Our results reveal a stark performance gap, shedding further light on the Corpus Bias problem, and the extent to which it continues to trouble these systems.