Classification by Attention: Scene Graph Classification with Prior Knowledge
This work offers a significant improvement in scene graph classification accuracy for computer vision researchers, particularly in low-data regimes.
This paper tackles the challenge of scene graph classification where object and relation appearances vary greatly across images. The authors propose a multi-task learning approach that integrates prior knowledge directly into an attention layer within the perception model, achieving over 3% improvement in object classification, 26% in scene graph classification, and 36% in predicate prediction accuracy with only 1% annotated images when combined with self-supervised learning.
A major challenge in scene graph classification is that the appearance of objects and relations can be significantly different from one image to another. Previous works have addressed this by relational reasoning over all objects in an image or incorporating prior knowledge into classification. Unlike previous works, we do not consider separate models for perception and prior knowledge. Instead, we take a multi-task learning approach, where we implement the classification as an attention layer. This allows for the prior knowledge to emerge and propagate within the perception model. By enforcing the model also to represent the prior, we achieve a strong inductive bias. We show that our model can accurately generate commonsense knowledge and that the iterative injection of this knowledge to scene representations leads to significantly higher classification performance. Additionally, our model can be fine-tuned on external knowledge given as triples. When combined with self-supervised learning and with 1% of annotated images only, this gives more than 3% improvement in object classification, 26% in scene graph classification, and 36% in predicate prediction accuracy.