Energy-Based Learning for Scene Graph Generation
This work addresses the challenge of structured prediction in scene graph generation for computer vision applications, offering an incremental improvement over existing methods.
The paper tackles the problem of scene graph generation by introducing an energy-based learning framework that incorporates output structure, leading to performance improvements of up to 21% on Visual Genome and 27% on GQA datasets.
Traditional scene graph generation methods are trained using cross-entropy losses that treat objects and relationships as independent entities. Such a formulation, however, ignores the structure in the output space, in an inherently structured prediction problem. In this work, we introduce a novel energy-based learning framework for generating scene graphs. The proposed formulation allows for efficiently incorporating the structure of scene graphs in the output space. This additional constraint in the learning framework acts as an inductive bias and allows models to learn efficiently from a small number of labels. We use the proposed energy-based framework to train existing state-of-the-art models and obtain a significant performance improvement, of up to 21% and 27%, on the Visual Genome and GQA benchmark datasets, respectively. Furthermore, we showcase the learning efficiency of the proposed framework by demonstrating superior performance in the zero- and few-shot settings where data is scarce.