Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning
This addresses the challenge of visual reasoning generalization for AI systems, particularly for unseen objects, though it is an incremental improvement based on active vision theories.
The paper tackled the problem of AI systems struggling with visual reasoning for unseen objects by developing a Glimpse-based Active Perception (GAP) system that sequentially fixates on salient image regions, resulting in state-of-the-art performance with improved sample efficiency and generalization to out-of-distribution inputs.
Human capabilities in understanding visual relations are far superior to those of AI systems, especially for previously unseen objects. For example, while AI systems struggle to determine whether two such objects are visually the same or different, humans can do so with ease. Active vision theories postulate that the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. In particular, the low-dimensional spatial information about the corresponding eye movements is hypothesized to facilitate the representation of relations between different image parts. Inspired by these theories, we develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Importantly, our system leverages the locations stemming from the glimpsing actions, along with the visual content around them, to represent relations between different parts of the image. The results suggest that the GAP is essential for extracting visual relations that go beyond the immediate visual content. Our approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.