An Interpretable Model for Scene Graph Generation
This work addresses the need for accurate and interpretable scene graph generation, which is incremental as it builds on existing methods but offers specific improvements for vision-language tasks like image captioning and visual QA.
The authors tackled scene graph generation by proposing an interpretable model that uses visual, spatial, and semantic features with late fusion, achieving a 5% absolute (20% relative) improvement over the second-place entry in the OpenImages Visual Relationship Detection Challenge.
We propose an efficient and interpretable scene graph generator. We consider three types of features: visual, spatial and semantic, and we use a late fusion strategy such that each feature's contribution can be explicitly investigated. We study the key factors about these features that have the most impact on the performance, and also visualize the learned visual features for relationships and investigate the efficacy of our model. We won the champion of the OpenImages Visual Relationship Detection Challenge on Kaggle, where we outperform the 2nd place by 5\% (20\% relatively). We believe an accurate scene graph generator is a fundamental stepping stone for higher-level vision-language tasks such as image captioning and visual QA, since it provides a semantic, structured comprehension of an image that is beyond pixels and objects.