The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
This work addresses the problem of understanding model performance in visual QA for researchers, revealing that minor architectural choices, rather than complex reasoning, drive success, which is incremental but clarifies design principles.
The paper investigates why simple feed-forward models perform well on the CLEVR visual QA dataset, finding that early fusion of language and vision significantly boosts performance, with a proposed Multimodal Core module achieving strong results.
Visual QA is a pivotal challenge for higher-level reasoning, requiring understanding language, vision, and relationships between many objects in a scene. Although datasets like CLEVR are designed to be unsolvable without such complex relational reasoning, some surprisingly simple feed-forward, "holistic" models have recently shown strong performance on this dataset. These models lack any kind of explicit iterative, symbolic reasoning procedure, which are hypothesized to be necessary for counting objects, narrowing down the set of relevant objects based on several attributes, etc. The reason for this strong performance is poorly understood. Hence, our work analyzes such models, and finds that minor architectural elements are crucial to performance. In particular, we find that \textit{early fusion} of language and vision provides large performance improvements. This contrasts with the late fusion approaches popular at the dawn of Visual QA. We propose a simple module we call Multimodal Core, which we hypothesize performs the fundamental operations for multimodal tasks. We believe that understanding why these elements are so important to complex question answering will aid the design of better-performing algorithms for Visual QA while minimizing hand-engineering effort.