MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling
This addresses the challenge of generalising piano performance synthesis across diverse MIDI sources and styles, though it is incremental as it adapts an existing TTS framework.
The paper tackles the problem of generating expressive piano audio performances from music scores by proposing MIDI-VALLE, a neural codec language model that conditions on reference audio and MIDI, achieving over 75% lower Frechet Audio Distance and receiving 202 vs. 58 votes in listening tests compared to a state-of-the-art baseline.
Generating expressive audio performances from music scores requires models to capture both instrument acoustics and human interpretation. Traditional music performance synthesis pipelines follow a two-stage approach, first generating expressive performance MIDI from a score, then synthesising the MIDI into audio. However, the synthesis models often struggle to generalise across diverse MIDI sources, musical styles, and recording environments. To address these challenges, we propose MIDI-VALLE, a neural codec language model adapted from the VALLE framework, which was originally designed for zero-shot personalised text-to-speech (TTS) synthesis. For performance MIDI-to-audio synthesis, we improve the architecture to condition on a reference audio performance and its corresponding MIDI. Unlike previous TTS-based systems that rely on piano rolls, MIDI-VALLE encodes both MIDI and audio as discrete tokens, facilitating a more consistent and robust modelling of piano performances. Furthermore, the model's generalisation ability is enhanced by training on an extensive and diverse piano performance dataset. Evaluation results show that MIDI-VALLE significantly outperforms a state-of-the-art baseline, achieving over 75% lower Frechet Audio Distance on the ATEPP and Maestro datasets. In the listening test, MIDI-VALLE received 202 votes compared to 58 for the baseline, demonstrating improved synthesis quality and generalisation across diverse performance MIDI inputs.