From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation
This work addresses the problem of generating structured symbolic music for applications in music composition, particularly for complex multi-track data, but it is incremental as it adapts existing NLP methods to a new domain.
The study investigated subword tokenization techniques like BPE and Unigram for symbolic music generation, finding that they enable longer song generation and improve structural metrics such as structure indicator and Pitch Class Entropy across various MIDI datasets.
Subword tokenization has been widely successful in text-based natural language processing (NLP) tasks with Transformer-based models. As Transformer models become increasingly popular in symbolic music-related studies, it is imperative to investigate the efficacy of subword tokenization in the symbolic music domain. In this paper, we explore subword tokenization techniques, such as byte-pair encoding (BPE), in symbolic music generation and its impact on the overall structure of generated songs. Our experiments are based on three types of MIDI datasets: single track-melody only, multi-track with a single instrument, and multi-track and multi-instrument. We apply subword tokenization on post-musical tokenization schemes and find that it enables the generation of longer songs at the same time and improves the overall structure of the generated music in terms of objective metrics like structure indicator (SI), Pitch Class Entropy, etc. We also compare two subword tokenization methods, BPE and Unigram, and observe that both methods lead to consistent improvements. Our study suggests that subword tokenization is a promising technique for symbolic music generation and may have broader implications for music composition, particularly in cases involving complex data such as multi-track songs.