Uncovering Temporal Context for Video Question and Answering
This addresses the problem of temporal understanding in videos for AI systems, though it appears incremental with hybrid methods.
The paper tackles video question answering by introducing a temporal domain approach to infer past, describe present, and predict future events, using an encoder-decoder RNN with dual-channel ranking loss, and demonstrates significant performance improvements over baselines on a dataset of 109,895 video clips and 390,744 questions.
In this work, we introduce Video Question Answering in temporal domain to infer the past, describe the present and predict the future. We present an encoder-decoder approach using Recurrent Neural Networks to learn temporal structures of videos and introduce a dual-channel ranking loss to answer multiple-choice questions. We explore approaches for finer understanding of video content using question form of "fill-in-the-blank", and managed to collect 109,895 video clips with duration over 1,000 hours from TACoS, MPII-MD, MEDTest 14 datasets, while the corresponding 390,744 questions are generated from annotations. Extensive experiments demonstrate that our approach significantly outperforms the compared baselines.