STEPs-RL: Speech-Text Entanglement for Phonetically Sound Representation Learning
This work addresses the problem of learning robust spoken-word representations for speech processing applications by incorporating both acoustic and semantic information, offering an incremental improvement over existing methods.
This paper introduces STEPs-RL, a multi-modal deep neural network that entangles speech and text to learn phonetically sound spoken-word representations. The model achieved 89.47% accuracy in predicting target phonetic sequences and competitive results against Word2Vec and FastText on word similarity benchmarks.
In this paper, we present a novel multi-modal deep neural network architecture that uses speech and text entanglement for learning phonetically sound spoken-word representations. STEPs-RL is trained in a supervised manner to predict the phonetic sequence of a target spoken-word using its contextual spoken word's speech and text, such that the model encodes its meaningful latent representations. Unlike existing work, we have used text along with speech for auditory representation learning to capture semantical and syntactical information along with the acoustic and temporal information. The latent representations produced by our model were not only able to predict the target phonetic sequences with an accuracy of 89.47% but were also able to achieve competitive results to textual word representation models, Word2Vec & FastText (trained on textual transcripts), when evaluated on four widely used word similarity benchmark datasets. In addition, investigation of the generated vector space also demonstrated the capability of the proposed model to capture the phonetic structure of the spoken-words. To the best of our knowledge, none of the existing works use speech and text entanglement for learning spoken-word representation, which makes this work first of its kind.