Improving Scene Graph Classification by Exploiting Knowledge from Texts
This work addresses the problem of data scarcity for training scene graph classification models, offering a method to significantly reduce the need for labor-intensive image annotations for researchers and developers in computer vision.
This paper explores using textual scene descriptions to train scene graph classification models, aiming to reduce reliance on annotated image data. By fine-tuning a classification pipeline with knowledge extracted from texts, the authors achieved approximately 8x more accurate results in scene graph classification, 3x in object classification, and 1.5x in predicate classification, using only 1% of the annotated images compared to supervised baselines.
Training scene graph classification models requires a large amount of annotated image data. Meanwhile, scene graphs represent relational knowledge that can be modeled with symbolic data from texts or knowledge graphs. While image annotation demands extensive labor, collecting textual descriptions of natural scenes requires less effort. In this work, we investigate whether textual scene descriptions can substitute for annotated image data. To this end, we employ a scene graph classification framework that is trained not only from annotated images but also from symbolic data. In our architecture, the symbolic entities are first mapped to their correspondent image-grounded representations and then fed into the relational reasoning pipeline. Even though a structured form of knowledge, such as the form in knowledge graphs, is not always available, we can generate it from unstructured texts using a transformer-based language model. We show that by fine-tuning the classification pipeline with the extracted knowledge from texts, we can achieve ~8x more accurate results in scene graph classification, ~3x in object classification, and ~1.5x in predicate classification, compared to the supervised baselines with only 1% of the annotated images.