VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images
This work addresses the problem of scaling VSAs for visual question answering on natural images, which is incremental as it extends existing VSA methods to more complex tasks.
The authors tackled the challenge of applying Vector Symbolic Architectures (VSAs) to complex spatial queries on natural images by proposing VSA4VQA, a 4D implementation that encodes objects in a hyperdimensional space with dimensions for width, height, and location, achieving competitive zero-shot performance on the GQA benchmark dataset.
While Vector Symbolic Architectures (VSAs) are promising for modelling spatial cognition, their application is currently limited to artificially generated images and simple spatial queries. We propose VSA4VQA - a novel 4D implementation of VSAs that implements a mental representation of natural images for the challenging task of Visual Question Answering (VQA). VSA4VQA is the first model to scale a VSA to complex spatial queries. Our method is based on the Semantic Pointer Architecture (SPA) to encode objects in a hyperdimensional vector space. To encode natural images, we extend the SPA to include dimensions for object's width and height in addition to their spatial location. To perform spatial queries we further introduce learned spatial query masks and integrate a pre-trained vision-language model for answering attribute-related questions. We evaluate our method on the GQA benchmark dataset and show that it can effectively encode natural images, achieving competitive performance to state-of-the-art deep learning methods for zero-shot VQA.