Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation
This work addresses a specific bottleneck in music sequence modeling for researchers and practitioners, offering an incremental improvement over existing compound token methods.
The paper tackles the problem of suboptimal performance in predicting compound tokens for symbolic music and audio generation by introducing the Nested Music Transformer (NMT), which decodes tokens sequentially to capture interdependencies, resulting in improved perplexity on datasets like MAESTRO.
Representing symbolic music with compound tokens, where each token consists of several different sub-tokens representing a distinct musical feature or attribute, offers the advantage of reducing sequence length. While previous research has validated the efficacy of compound tokens in music sequence modeling, predicting all sub-tokens simultaneously can lead to suboptimal results as it may not fully capture the interdependencies between them. We introduce the Nested Music Transformer (NMT), an architecture tailored for decoding compound tokens autoregressively, similar to processing flattened tokens, but with low memory usage. The NMT consists of two transformers: the main decoder that models a sequence of compound tokens and the sub-decoder for modeling sub-tokens of each compound token. The experiment results showed that applying the NMT to compound tokens can enhance the performance in terms of better perplexity in processing various symbolic music datasets and discrete audio tokens from the MAESTRO dataset.