Generative Spoken Language Modeling from Raw Audio
This addresses the challenge of unsupervised speech processing for applications where text data is unavailable, though it is incremental as it builds on existing encoders.
The paper tackles the problem of learning language from raw audio without text or labels by introducing Generative Spoken Language Modeling, with results showing that the number of discrete units affects performance in a task-dependent way and some combinations approach text-based systems.
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo-text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder-dependent way, and that some combinations approach text-based systems.