Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
This work addresses the challenge of capturing variable-length semantic flows in videos for improved video understanding, representing an incremental advance by introducing a novel graph-based method to handle complex dependencies beyond first-order Markovian models.
The authors tackled the problem of learning complex dependency structures in sequential video data, which conventional methods like RNNs struggle with, by proposing Cut-Based Graph Learning Networks (CB-GLNs) that represent videos as graphs and discover compositional dependencies, achieving the highest performance on video theme classification and video question answering tasks compared to baselines.
Conventional sequential learning methods such as Recurrent Neural Networks (RNNs) focus on interactions between consecutive inputs, i.e. first-order Markovian dependency. However, most of sequential data, as seen with videos, have complex dependency structures that imply variable-length semantic flows and their compositions, and those are hard to be captured by conventional methods. Here, we propose Cut-Based Graph Learning Networks (CB-GLNs) for learning video data by discovering these complex structures of the video. The CB-GLNs represent video data as a graph, with nodes and edges corresponding to frames of the video and their dependencies respectively. The CB-GLNs find compositional dependencies of the data in multilevel graph forms via a parameterized kernel with graph-cut and a message passing framework. We evaluate the proposed method on the two different tasks for video understanding: Video theme classification (Youtube-8M dataset) and Video Question and Answering (TVQA dataset). The experimental results show that our model efficiently learns the semantic compositional structure of video data. Furthermore, our model achieves the highest performance in comparison to other baseline methods.