Visual Commonsense based Heterogeneous Graph Contrastive Learning
This work addresses key bottlenecks in multi-modality applications like VQA, though it appears incremental as a plug-and-play enhancement to existing methods.
The paper tackles the problem of selecting relevant objects and reasoning about cross-domain relationships in visual question answering by incorporating visual commonsense information through heterogeneous graph contrastive learning. The method improves seven representative VQA models across four benchmarks, demonstrating significant performance gains.
How to select relevant key objects and reason about the complex relationships cross vision and linguistic domain are two key issues in many multi-modality applications such as visual question answering (VQA). In this work, we incorporate the visual commonsense information and propose a heterogeneous graph contrastive learning method to better finish the visual reasoning task. Our method is designed as a plug-and-play way, so that it can be quickly and easily combined with a wide range of representative methods. Specifically, our model contains two key components: the Commonsense-based Contrastive Learning and the Graph Relation Network. Using contrastive learning, we guide the model concentrate more on discriminative objects and relevant visual commonsense attributes. Besides, thanks to the introduction of the Graph Relation Network, the model reasons about the correlations between homogeneous edges and the similarities between heterogeneous edges, which makes information transmission more effective. Extensive experiments on four benchmarks show that our method greatly improves seven representative VQA models, demonstrating its effectiveness and generalizability.