Learning Joint Acoustic-Phonetic Word Embeddings
This work addresses speech recognition tasks by enabling better alignment of acoustic and phonetic word representations, though it is incremental in nature.
The paper tackles the problem of mapping words across acoustic and orthographic modalities by learning joint embeddings in a shared latent space, achieving an F1 score of 0.95 for a binary classification task.
Most speech recognition tasks pertain to mapping words across two modalities: acoustic and orthographic. In this work, we suggest learning encoders that map variable-length, acoustic or phonetic, sequences that represent words into fixed-dimensional vectors in a shared latent space; such that the distance between two word vectors represents how closely the two words sound. Instead of directly learning the distances between word vectors, we employ weak supervision and model a binary classification task to predict whether two inputs, one of each modality, represent the same word given a distance threshold. We explore various deep-learning models, bimodal contrastive losses, and techniques for mining hard negative examples such as the semi-supervised technique of self-labeling. Our best model achieves an $F_1$ score of 0.95 for the binary classification task.