Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
This addresses the problem of learning effective speech representations with minimal labeled data for tasks such as phoneme and speaker recognition, offering a novel approach that is not incremental.
The paper tackles unsupervised speech representation learning by introducing Mockingjay, a bidirectional Transformer encoder pre-trained on unlabeled speech, which predicts current frames using both past and future contexts. It improves performance on downstream tasks like phoneme classification and speaker recognition, achieving results that outperform Mel-features with only 0.1% labeled data compared to using 100% labeled data.
We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.