Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
This work addresses the challenge of generating high-quality audio for applications like media production and assistive technologies, offering a novel approach that improves efficiency and performance over existing methods.
The paper tackles audio generation by extending causal language models to continuous-valued tokens using token-wise diffusion and masked next-token prediction, achieving up to 40% relative gains in metrics like FAD and KL divergence over prior discrete methods and matching SOTA diffusion models with fewer parameters.
Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models.