Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings
This work addresses a limitation in AWE models for speech technology and auditory lexical processing by integrating lexical knowledge, though it is incremental as it builds on existing neural network approaches.
The paper tackled the problem of acoustic word embeddings (AWEs) not leveraging high-level lexical knowledge during training by proposing a multi-task learning model that incorporates top-down lexical information, such as word semantics, alongside bottom-up acoustic supervision. The result showed improved embedding space discriminability and better separation of lexical categories across three languages.
Models of acoustic word embeddings (AWEs) learn to map variable-length spoken word segments onto fixed-dimensionality vector representations such that different acoustic exemplars of the same word are projected nearby in the embedding space. In addition to their speech technology applications, AWE models have been shown to predict human performance on a variety of auditory lexical processing tasks. Current AWE models are based on neural networks and trained in a bottom-up approach that integrates acoustic cues to build up a word representation given an acoustic or symbolic supervision signal. Therefore, these models do not leverage or capture high-level lexical knowledge during the learning process. In this paper, we propose a multi-task learning model that incorporates top-down lexical knowledge into the training procedure of AWEs. Our model learns a mapping between the acoustic input and a lexical representation that encodes high-level information such as word semantics in addition to bottom-up form-based supervision. We experiment with three languages and demonstrate that incorporating lexical knowledge improves the embedding space discriminability and encourages the model to better separate lexical categories.