Generative Pre-Training for Speech with Autoregressive Predictive Coding
This work addresses the problem of learning transferable speech representations for various downstream tasks, offering incremental improvements in self-supervised learning for speech.
The paper tackles the challenge of learning general speech representations from unlabeled data by proposing autoregressive predictive coding (APC) as a generative pre-training method, achieving performance improvements over baseline features and other methods on speech recognition, translation, and speaker identification tasks, with effectiveness in reducing labeled data and model parameters.
Learning meaningful and general representations from unannotated speech that are applicable to a wide range of tasks remains challenging. In this paper we propose to use autoregressive predictive coding (APC), a recently proposed self-supervised objective, as a generative pre-training approach for learning meaningful, non-specific, and transferable speech representations. We pre-train APC on large-scale unlabeled data and conduct transfer learning experiments on three speech applications that require different information about speech characteristics to perform well: speech recognition, speech translation, and speaker identification. Extensive experiments show that APC not only outperforms surface features (e.g., log Mel spectrograms) and other popular representation learning methods on all three tasks, but is also effective at reducing downstream labeled data size and model parameters. We also investigate the use of Transformers for modeling APC and find it superior to RNNs.