WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation
This work addresses the challenge of extending single-stream autoregressive language modeling to speech, offering a simpler and more efficient approach for speech generation and modeling.
This paper introduces WavSLM, a speech language model that distills self-supervised WavLM representations into a single codebook and uses an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information in a single token stream without text supervision, achieving competitive performance on consistency benchmarks and speech generation with fewer parameters and less training data.
Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to speech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference. Demo samples are available at https://lucadellalib.github.io/wavslm-web/.