Barlow constrained optimization for Visual Question Answering
This work addresses the challenge of learning more informative joint representations for VQA, an incremental advance in multimodal AI with potential benefits for interpretability.
The paper tackles the problem of improving the informativeness of joint embedding spaces in Visual Question Answering by proposing a novel regularization method called Constrained Optimization using Barlow's theory (COB), which reduces redundancy and aligns spaces, resulting in accuracy improvements of 1.4% on VQA-CP v2 and 4% on VQA v2 datasets.
Visual question answering is a vision-and-language multimodal task, that aims at predicting answers given samples from the question and image modalities. Most recent methods focus on learning a good joint embedding space of images and questions, either by improving the interaction between these two modalities, or by making it a more discriminant space. However, how informative this joint space is, has not been well explored. In this paper, we propose a novel regularization for VQA models, Constrained Optimization using Barlow's theory (COB), that improves the information content of the joint space by minimizing the redundancy. It reduces the correlation between the learned feature components and thereby disentangles semantic concepts. Our model also aligns the joint space with the answer embedding space, where we consider the answer and image+question as two different `views' of what in essence is the same semantic information. We propose a constrained optimization policy to balance the categorical and redundancy minimization forces. When built on the state-of-the-art GGE model, the resulting model improves VQA accuracy by 1.4% and 4% on the VQA-CP v2 and VQA v2 datasets respectively. The model also exhibits better interpretability.