3D-Aware Visual Question Answering about Parts, Poses and Occlusions
This addresses the limitation of current VQA models that focus only on 2D reasoning, which is important for applications like navigation and manipulation where 3D understanding is crucial.
The paper tackles the problem of enabling visual question answering models to understand 3D scene structure by introducing the task of 3D-aware VQA, which requires compositional reasoning about object parts, poses, and occlusions. The authors propose a new dataset (Super-CLEVR-3D) and model (PO3D-VQA) that significantly outperforms existing methods, though a performance gap remains compared to 2D VQA benchmarks.
Despite rapid progress in Visual question answering (VQA), existing datasets and models mainly focus on testing reasoning in 2D. However, it is important that VQA models also understand the 3D structure of visual scenes, for example to support tasks like navigation or manipulation. This includes an understanding of the 3D object pose, their parts and occlusions. In this work, we introduce the task of 3D-aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes. We address 3D-aware VQA from both the dataset and the model perspective. First, we introduce Super-CLEVR-3D, a compositional reasoning dataset that contains questions about object parts, their 3D poses, and occlusions. Second, we propose PO3D-VQA, a 3D-aware VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and deep neural networks with 3D generative representations of objects for robust visual recognition. Our experimental results show our model PO3D-VQA outperforms existing methods significantly, but we still observe a significant performance gap compared to 2D VQA benchmarks, indicating that 3D-aware VQA remains an important open research area.