SG-Shuffle: Multi-aspect Shuffle Transformer for Scene Graph Generation
This work addresses biased and inaccurate scene graph generation for visual understanding tasks, representing an incremental improvement over existing methods.
The paper tackles the long-tail bias and non-mutually exclusive relationship challenges in Scene Graph Generation (SGG) by proposing SG-Shuffle, a pipeline with a Parallel Transformer Encoder, Shuffle Transformer, and Weighted CE loss, achieving improved prediction of suitable relationship labels.
Scene Graph Generation (SGG) serves a comprehensive representation of the images for human understanding as well as visual understanding tasks. Due to the long tail bias problem of the object and predicate labels in the available annotated data, the scene graph generated from current methodologies can be biased toward common, non-informative relationship labels. Relationship can sometimes be non-mutually exclusive, which can be described from multiple perspectives like geometrical relationships or semantic relationships, making it even more challenging to predict the most suitable relationship label. In this work, we proposed the SG-Shuffle pipeline for scene graph generation with 3 components: 1) Parallel Transformer Encoder, which learns to predict object relationships in a more exclusive manner by grouping relationship labels into groups of similar purpose; 2) Shuffle Transformer, which learns to select the final relationship labels from the category-specific feature generated in the previous step; and 3) Weighted CE loss, used to alleviate the training bias caused by the imbalanced dataset.