DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation
This addresses the problem of generating high-quality vocal-accompaniment music from lyrics for AI music systems, representing an incremental improvement by optimizing tokenization for dual-track structure.
The paper tackles the trade-off between reconstruction quality and language-model learnability in lyrics-to-song systems by introducing DUO-TOK, a dual-track tokenizer for vocal-accompaniment music, which achieves the best music-tagging AP and lowest vocabulary-normalized LM perplexity at 0.75 kbps while maintaining comparable reconstruction quality to state-of-the-art tokenizers.
Duo-Tok is a source-aware dual-codebook tokenizer for vocal-accompaniment music that targets the growing tension between reconstruction quality and language-model (LM) learnability in modern lyrics-to-song systems. Existing codecs either prioritize high-fidelity reconstruction with difficult-to-model acoustic tokens or compress aggressively into semantic tokens that are LM-friendly but lossy, and they rarely make the tokenizer itself aware of dual-track structure. Duo-Tok follows a four-stage, SSL-centered pipeline: we first pretrain a BEST-RQ-style encoder on large-scale audio, then stabilize and factorize the representation with Gaussian replacement noise and multi-task supervision, before freezing the encoder to learn SimVQ-based dual codebooks with hard routing for vocals and accompaniment, and finally training latent diffusion decoders on top of the discrete tokens. Duo-Tok at 0.75 kbps shifts the empirical reconstruction-generation Pareto frontier, achieving the best music-tagging AP and the lowest vocabulary-normalized LM perplexity among compared codecs while maintaining reconstruction quality comparable to state-of-the-art music tokenizers.