Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder
This addresses the problem of automated classical music performance generation for applications in music AI and creative tools, representing an incremental advance by combining existing methods like VQ-VAE and VAE in a novel architecture.
The paper tackled generating classical piano performances from scratch by emulating both composition and expressive interpretation, introducing the Expressive Compound Word representation and the Expressive Music Variational AutoEncoder model, which achieved superior musical quality compared to state-of-the-art models in evaluations.
The creativity of classical music arises not only from composers who craft the musical sheets but also from performers who interpret the static notations with expressive nuances. This paper addresses the challenge of generating classical piano performances from scratch, aiming to emulate the dual roles of composer and pianist in the creative process. We introduce the Expressive Compound Word (ECP) representation, which effectively captures both the metrical structure and expressive nuances of classical performances. Building on this, we propose the Expressive Music Variational AutoEncoder (XMVAE), a model featuring two branches: a Vector Quantized Variational AutoEncoder (VQ-VAE) branch that generates score-related content, representing the Composer, and a vanilla VAE branch that produces expressive details, fulfilling the role of Pianist. These branches are jointly trained with similar Seq2Seq architectures, leveraging a multiscale encoder to capture beat-level contextual information and an orthogonal Transformer decoder for efficient compound tokens decoding. Both objective and subjective evaluations demonstrate that XMVAE generates classical performances with superior musical quality compared to state-of-the-art models. Furthermore, pretraining the Composer branch on extra musical score datasets contribute to a significant performance gain.