Estimating semantic structure for the VQA answer space
This work addresses the semantic limitations in VQA models for researchers and practitioners, offering an incremental improvement by integrating proximity-based methods with existing approaches.
The authors tackled the problem of Visual Question Answering (VQA) being treated as a classification task with independent answers, which ignores semantic similarities between classes. They proposed proximity measures and a corresponding loss that improved generalization and reduced language bias, achieving state-of-the-art performance on the VQAv2-CP dataset.
Since its appearance, Visual Question Answering (VQA, i.e. answering a question posed over an image), has always been treated as a classification problem over a set of predefined answers. Despite its convenience, this classification approach poorly reflects the semantics of the problem limiting the answering to a choice between independent proposals, without taking into account the similarity between them (e.g. equally penalizing for answering cat or German shepherd instead of dog). We address this issue by proposing (1) two measures of proximity between VQA classes, and (2) a corresponding loss which takes into account the estimated proximity. This significantly improves the generalization of VQA models by reducing their language bias. In particular, we show that our approach is completely model-agnostic since it allows consistent improvements with three different VQA models. Finally, by combining our method with a language bias reduction approach, we report SOTA-level performance on the challenging VQAv2-CP dataset.