End-to-End Multimodal Representation Learning for Video Dialog
This work addresses the challenge of effectively utilizing visual features in multimodal learning for video dialog, which is an incremental improvement over existing methods.
The paper tackled the problem of video-based dialog by addressing the bias of existing models toward textual information over visual cues, proposing a new framework that combines 3D-CNN and transformer networks into a single visual encoder to extract more robust semantic representations from videos, resulting in significant improvements over baselines in generative and retrieval tasks on the AVSD task.
Video-based dialog task is a challenging multimodal learning task that has received increasing attention over the past few years with state-of-the-art obtaining new performance records. This progress is largely powered by the adaptation of the more powerful transformer-based language encoders. Despite this progress, existing approaches do not effectively utilize visual features to help solve tasks. Recent studies show that state-of-the-art models are biased toward textual information rather than visual cues. In order to better leverage the available visual information, this study proposes a new framework that combines 3D-CNN network and transformer-based networks into a single visual encoder to extract more robust semantic representations from videos. The visual encoder is jointly trained end-to-end with other input modalities such as text and audio. Experiments on the AVSD task show significant improvement over baselines in both generative and retrieval tasks.