Unspeech: Unsupervised Speech Context Embeddings
This provides a method for unsupervised speech representation learning, potentially benefiting speech processing tasks, but it is incremental as it builds on existing context discrimination techniques.
The paper tackled the problem of learning speech context embeddings without supervision, introducing Unspeech embeddings trained on 9500 hours of unlabeled English speech, which achieved consistent word error rate reductions on out-of-domain data compared to i-vector baselines.
We introduce "Unspeech" embeddings, which are based on unsupervised learning of context feature representations for spoken language. The embeddings were trained on up to 9500 hours of crawled English speech data without transcriptions or speaker information, by using a straightforward learning objective based on context and non-context discrimination with negative sampling. We use a Siamese convolutional neural network architecture to train Unspeech embeddings and evaluate them on speaker comparison, utterance clustering and as a context feature in TDNN-HMM acoustic models trained on TED-LIUM, comparing it to i-vector baselines. Particularly decoding out-of-domain speech data from the recently released Common Voice corpus shows consistent WER reductions. We release our source code and pre-trained Unspeech models under a permissive open source license.