SAViR-T: Spatially Attentive Visual Reasoning with Transformers
This work addresses visual reasoning challenges for AI systems, representing an incremental improvement with strong performance gains.
The authors tackled visual reasoning problems in Raven's Progressive Matrices by developing SAViR-T, a transformer-based model that uses spatially attentive tokens to learn intra- and inter-image dependencies, achieving new state-of-the-art results across multiple benchmarks.
We present a novel computational model, "SAViR-T", for the family of visual reasoning problems embodied in the Raven's Progressive Matrices (RPM). Our model considers explicit spatial semantics of visual elements within each image in the puzzle, encoded as spatio-visual tokens, and learns the intra-image as well as the inter-image token dependencies, highly relevant for the visual reasoning task. Token-wise relationship, modeled through a transformer-based SAViR-T architecture, extract group (row or column) driven representations by leveraging the group-rule coherence and use this as the inductive bias to extract the underlying rule representations in the top two row (or column) per token in the RPM. We use this relation representations to locate the correct choice image that completes the last row or column for the RPM. Extensive experiments across both synthetic RPM benchmarks, including RAVEN, I-RAVEN, RAVEN-FAIR, and PGM, and the natural image-based "V-PROM" demonstrate that SAViR-T sets a new state-of-the-art for visual reasoning, exceeding prior models' performance by a considerable margin.