Compositional Structure Learning for Sequential Video Data
This addresses the challenge of modeling variable-length semantic flows in videos for computer vision applications, representing an incremental improvement over existing sequential learning methods.
The authors tackled the problem of capturing complex temporal dependencies in sequential video data, which conventional methods like RNNs struggle with, by proposing Temporal Dependency Networks (TDNs) that represent videos as graphs and use graph-cut and convolutions to discover compositional structures, achieving efficient learning on the Youtube-8M dataset.
Conventional sequential learning methods such as Recurrent Neural Networks (RNNs) focus on interactions between consecutive inputs, i.e. first-order Markovian dependency. However, most of sequential data, as seen with videos, have complex temporal dependencies that imply variable-length semantic flows and their compositions, and those are hard to be captured by conventional methods. Here, we propose Temporal Dependency Networks (TDNs) for learning video data by discovering these complex structures of the videos. The TDNs represent video as a graph whose nodes and edges correspond to frames of the video and their dependencies respectively. Via a parameterized kernel with graph-cut and graph convolutions, the TDNs find compositional temporal dependencies of the data in multilevel graph forms. We evaluate the proposed method on the large-scale video dataset Youtube-8M. The experimental results show that our model efficiently learns the complex semantic structure of video data.