A Picture May Be Worth a Hundred Words for Visual Question Answering
This addresses the challenge of detailed image understanding in VQA by simplifying the process and reducing computational cost, though it is incremental as it builds on existing language models and data augmentation techniques.
The paper tackles the problem of using textual representations instead of deep visual features for visual question answering (VQA), showing that descriptions with about a hundred words can compete with conventional features on VQA 2.0 and VQA-CP v2 benchmarks.
How far can we go with textual representations for understanding pictures? In image understanding, it is essential to use concise but detailed image representations. Deep visual features extracted by vision models, such as Faster R-CNN, are prevailing used in multiple tasks, and especially in visual question answering (VQA). However, conventional deep visual features may struggle to convey all the details in an image as we humans do. Meanwhile, with recent language models' progress, descriptive text may be an alternative to this problem. This paper delves into the effectiveness of textual representations for image understanding in the specific context of VQA. We propose to take description-question pairs as input, instead of deep visual features, and fed them into a language-only Transformer model, simplifying the process and the computational cost. We also experiment with data augmentation techniques to increase the diversity in the training set and avoid learning statistical bias. Extensive evaluations have shown that textual representations require only about a hundred words to compete with deep visual features on both VQA 2.0 and VQA-CP v2.