Detecting Visual Relationships Using Box Attention
This work addresses the need for structured image understanding beyond individual object detection, which is important for applications in computer vision, though it appears incremental by building on standard object detection pipelines without adding complex components.
The paper tackles the problem of detecting visual relationships in images, such as 'person riding motorcycle', by introducing a Box Attention mechanism that models pairwise interactions between objects. The model achieves strong quantitative and qualitative results on three challenging datasets, including V-COCO, Visual Relationships, and Open Images.
We propose a new model for detecting visual relationships, such as "person riding motorcycle" or "bottle on table". This task is an important step towards comprehensive structured image understanding, going beyond detecting individual objects. Our main novelty is a Box Attention mechanism that allows to model pairwise interactions between objects using standard object detection pipelines. The resulting model is conceptually clean, expressive and relies on well-justified training and prediction procedures. Moreover, unlike previously proposed approaches, our model does not introduce any additional complex components or hyperparameters on top of those already required by the underlying detection model. We conduct an experimental evaluation on three challenging datasets, V-COCO, Visual Relationships and Open Images, demonstrating strong quantitative and qualitative results.