Multitrack Music Transcription with a Time-Frequency Perceiver
This addresses the challenging problem of transcribing multiple instruments and vocals in music audio for music information retrieval applications, representing a domain-specific incremental improvement.
The paper tackles multitrack music transcription including vocals by proposing Perceiver TF, a novel deep neural network architecture that extends the Perceiver with hierarchical expansion and an additional Transformer layer for temporal coherence, achieving state-of-the-art performance on public datasets.
Multitrack music transcription aims to transcribe a music audio input into the musical notes of multiple instruments simultaneously. It is a very challenging task that typically requires a more complex model to achieve satisfactory result. In addition, prior works mostly focus on transcriptions of regular instruments, however, neglecting vocals, which are usually the most important signal source if present in a piece of music. In this paper, we propose a novel deep neural network architecture, Perceiver TF, to model the time-frequency representation of audio input for multitrack transcription. Perceiver TF augments the Perceiver architecture by introducing a hierarchical expansion with an additional Transformer layer to model temporal coherence. Accordingly, our model inherits the benefits of Perceiver that posses better scalability, allowing it to well handle transcriptions of many instruments in a single model. In experiments, we train a Perceiver TF to model 12 instrument classes as well as vocal in a multi-task learning manner. Our result demonstrates that the proposed system outperforms the state-of-the-art counterparts (e.g., MT3 and SpecTNT) on various public datasets.