Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation
This addresses the challenge of visualizing semantic concepts from text for applications in AI and human-computer interaction, but it is incremental as it builds on existing embedding and neural network techniques.
The paper tackles the problem of generating images from word embeddings, implementing a method that maps embeddings to visual representations and then to pixel space, with user studies indicating the system captures general visual properties like color and environment.
We introduce language-driven image generation, the task of generating an image visualizing the semantic contents of a word embedding, e.g., given the word embedding of grasshopper, we generate a natural image of a grasshopper. We implement a simple method based on two mapping functions. The first takes as input a word embedding (as produced, e.g., by the word2vec toolkit) and maps it onto a high-level visual space (e.g., the space defined by one of the top layers of a Convolutional Neural Network). The second function maps this abstract visual representation to pixel space, in order to generate the target image. Several user studies suggest that the current system produces images that capture general visual properties of the concepts encoded in the word embedding, such as color or typical environment, and are sufficient to discriminate between general categories of objects.