Learning Modality-Invariant Representations for Speech and Images
This addresses the challenge of cross-modality information retrieval and transfer learning, though it appears incremental as it builds on existing VAE techniques applied to a specific domain.
The paper tackles the problem of learning modality-invariant semantic representations for co-occurring speech and image data (spoken and handwritten digits from TIDIGITs and MNIST datasets), achieving this by mapping inputs to Gaussian distributions with a VAE-inspired regularization term that filters out modality-specific information while preserving semantics.
In this paper, we explore the unsupervised learning of a semantic embedding space for co-occurring sensory inputs. Specifically, we focus on the task of learning a semantic vector space for both spoken and handwritten digits using the TIDIGITs and MNIST datasets. Current techniques encode image and audio/textual inputs directly to semantic embeddings. In contrast, our technique maps an input to the mean and log variance vectors of a diagonal Gaussian from which sample semantic embeddings are drawn. In addition to encouraging semantic similarity between co-occurring inputs,our loss function includes a regularization term borrowed from variational autoencoders (VAEs) which drives the posterior distributions over embeddings to be unit Gaussian. We can use this regularization term to filter out modality information while preserving semantic information. We speculate this technique may be more broadly applicable to other areas of cross-modality/domain information retrieval and transfer learning.