Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
This addresses scalability and generalization issues in vision understanding for applications like robotics and image retrieval, though it builds incrementally on existing SGG methods.
The paper tackles the long-tail distribution problem in Scene Graph Generation where tail predicates have limited training data, proposing a Cross-modal prediCate boosting (CaCao) framework that uses a visually-prompted language model to generate diverse fine-grained predicates in a low-resource way, and shows it consistently boosts performance on three benchmark datasets while achieving competitive results in open-world predicate prediction.
Scene Graph Generation (SGG) aims to extract <subject, predicate, object> relationships in images for vision understanding. Although recent works have made steady progress on SGG, they still suffer long-tail distribution issues that tail-predicates are more costly to train and hard to distinguish due to a small amount of annotated data compared to frequent predicates. Existing re-balancing strategies try to handle it via prior rules but are still confined to pre-defined conditions, which are not scalable for various models and datasets. In this paper, we propose a Cross-modal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates in a low-resource way. The proposed CaCao can be applied in a plug-and-play fashion and automatically strengthen existing SGG to tackle the long-tailed problem. Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zero-shot manner. Comprehensive experiments on three benchmark datasets show that CaCao consistently boosts the performance of multiple scene graph generation models in a model-agnostic way. Moreover, our Epic achieves competitive performance on open-world predicate prediction. The data and code for this paper are publicly available.