Exploring High-Order Self-Similarity for Video Understanding
This work addresses the challenge of motion modeling in video understanding for applications like action recognition and robotics, offering a general temporal modeling module with broad applicability, though it appears incremental as it builds on existing STSS concepts.
The paper tackles the problem of representing temporal dynamics in video understanding by exploring higher-order space-time self-similarity (STSS) and introduces the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module that learns and integrates multi-order STSS features, resulting in substantial improvements across diverse video tasks with marginal computational cost.
Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.