STAR-GNN: Spatial-Temporal Video Representation for Content-based Retrieval
It addresses video retrieval for applications like media search, but appears incremental as it builds on graph neural networks and existing retrieval methods.
The paper tackles video representation learning for content-based retrieval by proposing STAR-GNN, a framework that models videos as multi-scale lattice graphs to capture spatial-temporal dynamics, achieving state-of-the-art performance in retrieval tasks.
We propose a video feature representation learning framework called STAR-GNN, which applies a pluggable graph neural network component on a multi-scale lattice feature graph. The essence of STAR-GNN is to exploit both the temporal dynamics and spatial contents as well as visual connections between regions at different scales in the frames. It models a video with a lattice feature graph in which the nodes represent regions of different granularity, with weighted edges that represent the spatial and temporal links. The contextual nodes are aggregated simultaneously by graph neural networks with parameters trained with retrieval triplet loss. In the experiments, we show that STAR-GNN effectively implements a dynamic attention mechanism on video frame sequences, resulting in the emphasis for dynamic and semantically rich content in the video, and is robust to noise and redundancies. Empirical results show that STAR-GNN achieves state-of-the-art performance for Content-Based Video Retrieval.