The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
This work establishes a new benchmark and task for researchers developing unsupervised spoken language models, providing a standardized way to evaluate progress in learning linguistic structures directly from speech. It is an incremental step towards fully unsupervised speech understanding.
This paper introduces a new unsupervised task, spoken language modeling, which involves learning linguistic representations from raw audio signals without labels. The authors also present the Zero Resource Speech Benchmark 2021, a suite of four metrics to evaluate model quality at phonetic, lexical, syntactic, and semantic levels. A composite baseline, combining contrastive representation learning, clustering, and language modeling, achieved better than chance performance on all four metrics, demonstrating the feasibility of the task.
We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics. We present the results and analyses of a composite baseline made of the concatenation of three unsupervised systems: self-supervised contrastive representation learning (CPC), clustering (k-means) and language modeling (LSTM or BERT). The language models learn on the basis of the pseudo-text derived from clustering the learned representations. This simple pipeline shows better than chance performance on all four metrics, demonstrating the feasibility of spoken language modeling from raw speech. It also yields worse performance compared to text-based 'topline' systems trained on the same data, delineating the space to be explored by more sophisticated end-to-end models.