Importance Weighted Structure Learning for Scene Graph Generation
This work addresses performance issues in scene graph generation, a key task for computer vision applications, but it is incremental as it builds on existing variational Bayesian methods.
The paper tackles the problem of variational inference underestimating the posterior in scene graph generation, leading to inferior performance, by proposing an importance weighted structure learning method that achieves state-of-the-art results on popular benchmarks.
Scene graph generation is a structured prediction task aiming to explicitly model objects and their relationships via constructing a visually-grounded scene graph for an input image. Currently, the message passing neural network based mean field variational Bayesian methodology is the ubiquitous solution for such a task, in which the variational inference objective is often assumed to be the classical evidence lower bound. However, the variational approximation inferred from such loose objective generally underestimates the underlying posterior, which often leads to inferior generation performance. In this paper, we propose a novel importance weighted structure learning method aiming to approximate the underlying log-partition function with a tighter importance weighted lower bound, which is computed from multiple samples drawn from a reparameterizable Gumbel-Softmax sampler. A generic entropic mirror descent algorithm is applied to solve the resulting constrained variational inference task. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.