Generating Compositional Color Representations from Text
This addresses a cross-modal task for image search engines, focusing on color composition in text queries, but it is incremental as it builds on existing GAN and contrastive learning methods.
The paper tackles the problem of generating color representations for text phrases, particularly (attribute, object) pairs, using a generative adversarial network that learns composition to combine seen elements into unseen pairs. It achieves lower Frechet Inception Distance than discriminative models, improving image retrieval and classification performance.
We consider the cross-modal task of producing color representations for text phrases. Motivated by the fact that a significant fraction of user queries on an image search engine follow an (attribute, object) structure, we propose a generative adversarial network that generates color profiles for such bigrams. We design our pipeline to learn composition - the ability to combine seen attributes and objects to unseen pairs. We propose a novel dataset curation pipeline from existing public sources. We describe how a set of phrases of interest can be compiled using a graph propagation technique, and then mapped to images. While this dataset is specialized for our investigations on color, the method can be extended to other visual dimensions where composition is of interest. We provide detailed ablation studies that test the behavior of our GAN architecture with loss functions from the contrastive learning literature. We show that the generative model achieves lower Frechet Inception Distance than discriminative ones, and therefore predicts color profiles that better match those from real images. Finally, we demonstrate improved performance in image retrieval and classification, indicating the crucial role that color plays in these downstream tasks.