Temporally Consistent Dynamic Scene Graphs: An End-to-End Approach for Action Tracklet Generation
This work addresses the problem of multi-frame video analysis for applications such as surveillance and autonomous navigation, representing a strong specific gain rather than a foundational advancement.
The paper tackles the challenge of capturing dynamic interactions in videos by introducing TCDSG, an end-to-end framework for generating temporally consistent action tracklets, achieving over 60% improvement in temporal recall@k on datasets like Action Genome, OpenPVSG, and MEVA.
Understanding video content is pivotal for advancing real-world applications like activity recognition, autonomous systems, and human-computer interaction. While scene graphs are adept at capturing spatial relationships between objects in individual frames, extending these representations to capture dynamic interactions across video sequences remains a significant challenge. To address this, we present TCDSG, Temporally Consistent Dynamic Scene Graphs, an innovative end-to-end framework that detects, tracks, and links subject-object relationships across time, generating action tracklets, temporally consistent sequences of entities and their interactions. Our approach leverages a novel bipartite matching mechanism, enhanced by adaptive decoder queries and feedback loops, ensuring temporal coherence and robust tracking over extended sequences. This method not only establishes a new benchmark by achieving over 60% improvement in temporal recall@k on the Action Genome, OpenPVSG, and MEVA datasets but also pioneers the augmentation of MEVA with persistent object ID annotations for comprehensive tracklet generation. By seamlessly integrating spatial and temporal dynamics, our work sets a new standard in multi-frame video analysis, opening new avenues for high-impact applications in surveillance, autonomous navigation, and beyond.