Visually Grounded Concept Composition
This addresses the challenge of robust visual grounding for natural language processing and computer vision applications, though it appears incremental as it builds on existing constituency analysis methods.
The paper tackles the problem of composing complex concepts in text from primitive ones while grounding them in images, resulting in improved text-to-image matching accuracy, especially when evaluation data diverges from training data.
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images. We propose Concept and Relation Graph (CRG), which builds on top of constituency analysis and consists of recursively combined concepts with predicate functions. Meanwhile, we propose a concept composition neural network called Composer to leverage the CRG for visually grounded concept learning. Specifically, we learn the grounding of both primitive and all composed concepts by aligning them to images and show that learning to compose leads to more robust grounding results, measured in text-to-image matching accuracy. Notably, our model can model grounded concepts forming at both the finer-grained sentence level and the coarser-grained intermediate level (or word-level). Composer leads to pronounced improvement in matching accuracy when the evaluation data has significant compound divergence from the training data.