MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation
This addresses the challenge of video-grounded dialogue generation for AI systems, offering a novel method to enhance spatio-temporal reasoning, though it appears incremental as it builds on existing encoder-decoder models.
The paper tackles the problem of generating dialogue grounded in videos by proposing MSG-BART, which integrates multi-granularity spatio-temporal scene graphs into an encoder-decoder model to improve visual reasoning; experiments on three benchmarks show significant superiority over state-of-the-art approaches.
Generating dialogue grounded in videos requires a high level of understanding and reasoning about the visual scenes in the videos. However, existing large visual-language models are not effective due to their latent features and decoder-only structure, especially with respect to spatio-temporal relationship reasoning. In this paper, we propose a novel approach named MSG-BART, which enhances the integration of video information by incorporating a multi-granularity spatio-temporal scene graph into an encoder-decoder pre-trained language model. Specifically, we integrate the global and local scene graph into the encoder and decoder, respectively, to improve both overall perception and target reasoning capability. To further improve the information selection capability, we propose a multi-pointer network to facilitate selection between text and video. Extensive experiments are conducted on three video-grounded dialogue benchmarks, which show the significant superiority of the proposed MSG-BART compared to a range of state-of-the-art approaches.