Temporal Alignment-Free Video Matching for Few-shot Action Recognition
This work addresses the problem of few-shot action recognition for video analysis, offering a novel method that improves flexibility and efficiency over existing alignment-based approaches.
The paper tackles the challenge of handling divergent narrative trajectories in Few-Shot Action Recognition by introducing TEAM, a temporal alignment-free matching approach that uses pattern tokens for flexible video representation, achieving state-of-the-art results on benchmarks like SSv2-Full and Kinetics.
Few-Shot Action Recognition (FSAR) aims to train a model with only a few labeled video instances. A key challenge in FSAR is handling divergent narrative trajectories for precise video matching. While the frame- and tuple-level alignment approaches have been promising, their methods heavily rely on pre-defined and length-dependent alignment units (e.g., frames or tuples), which limits flexibility for actions of varying lengths and speeds. In this work, we introduce a novel TEmporal Alignment-free Matching (TEAM) approach, which eliminates the need for temporal units in action representation and brute-force alignment during matching. Specifically, TEAM represents each video with a fixed set of pattern tokens that capture globally discriminative clues within the video instance regardless of action length or speed, ensuring its flexibility. Furthermore, TEAM is inherently efficient, using token-wise comparisons to measure similarity between videos, unlike existing methods that rely on pairwise comparisons for temporal alignment. Additionally, we propose an adaptation process that identifies and removes common information across classes, establishing clear boundaries even between novel categories. Extensive experiments demonstrate the effectiveness of TEAM. Codes are available at github.com/leesb7426/TEAM.