Learning Graph Embeddings for Compositional Zero-shot Learning
This paper tackles the problem of recognizing unseen visual compositions for computer vision systems, which is challenging due to the varying impact of states on different objects.
This paper addresses compositional zero-shot learning, where the goal is to recognize unseen combinations of visual states and objects. The authors propose Compositional Graph Embedding (CGE), an end-to-end model that learns image features, classifiers, and latent representations by exploiting dependencies within a graph structure. CGE significantly outperforms state-of-the-art methods on the MIT-States and UT-Zappos datasets.
In compositional zero-shot learning, the goal is to recognize unseen compositions (e.g. old dog) of observed visual primitives states (e.g. old, cute) and objects (e.g. car, dog) in the training set. This is challenging because the same state can for example alter the visual appearance of a dog drastically differently from a car. As a solution, we propose a novel graph formulation called Compositional Graph Embedding (CGE) that learns image features, compositional classifiers, and latent representations of visual primitives in an end-to-end manner. The key to our approach is exploiting the dependency between states, objects, and their compositions within a graph structure to enforce the relevant knowledge transfer from seen to unseen compositions. By learning a joint compatibility that encodes semantics between concepts, our model allows for generalization to unseen compositions without relying on an external knowledge base like WordNet. We show that in the challenging generalized compositional zero-shot setting our CGE significantly outperforms the state of the art on MIT-States and UT-Zappos. We also propose a new benchmark for this task based on the recent GQA dataset. Code is available at: https://github.com/ExplainableML/czsl