See, Hear, and Read: Deep Aligned Representations
This work addresses the challenge of multimodal integration for tasks like retrieval and classification, though it is incremental in building on existing deep learning methods.
The paper tackles the problem of learning a shared representation across vision, sound, and language by leveraging large-scale synchronous data, resulting in a representation that enables cross-modal retrieval and transfer, including between unseen text and sound pairs.
We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language. By leveraging over a year of sound from video and millions of sentences paired with images, we jointly train a deep convolutional network for aligned representation learning. Our experiments suggest that this representation is useful for several tasks, such as cross-modal retrieval or transferring classifiers between modalities. Moreover, although our network is only trained with image+text and image+sound pairs, it can transfer between text and sound as well, a transfer the network never observed during training. Visualizations of our representation reveal many hidden units which automatically emerge to detect concepts, independent of the modality.