Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
This work addresses efficiency limitations for real-time scene graph generation in computer vision applications, representing an incremental improvement over existing methods.
The paper tackles the problem of slow inference speed in scene graph generation by proposing a subgraph-based framework that factorizes the scene graph into subgraphs, reducing computation and maintaining spatial information. It outperforms state-of-the-art methods in accuracy and speed on Visual Relationship Detection and Visual Genome datasets.
Generating scene graph to describe all the relations inside an image gains increasing interests these years. However, most of the previous methods use complicated structures with slow inference speed or rely on the external data, which limits the usage of the model in real-life scenarios. To improve the efficiency of scene graph generation, we propose a subgraph-based connection graph to concisely represent the scene graph during the inference. A bottom-up clustering method is first used to factorize the entire scene graph into subgraphs, where each subgraph contains several objects and a subset of their relationships. By replacing the numerous relationship representations of the scene graph with fewer subgraph and object features, the computation in the intermediate stage is significantly reduced. In addition, spatial information is maintained by the subgraph features, which is leveraged by our proposed Spatial-weighted Message Passing~(SMP) structure and Spatial-sensitive Relation Inference~(SRI) module to facilitate the relationship recognition. On the recent Visual Relationship Detection and Visual Genome datasets, our method outperforms the state-of-the-art method in both accuracy and speed.