Temporal-Viewpoint Transportation Plan for Skeletal Few-shot Action Recognition
This work addresses action recognition for applications like surveillance or human-computer interaction, but it is incremental as it builds on existing few-shot and alignment techniques.
The paper tackles the problem of misalignment in 3D skeleton-based action recognition under few-shot learning by proposing a method that jointly aligns sequences in temporal and camera viewpoint spaces, achieving state-of-the-art results on datasets like NTU-60 and Kinetics-skeleton.
We propose a Few-shot Learning pipeline for 3D skeleton-based action recognition by Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE). To factor out misalignment between query and support sequences of 3D body joints, we propose an advanced variant of Dynamic Time Warping which jointly models each smooth path between the query and support frames to achieve simultaneously the best alignment in the temporal and simulated camera viewpoint spaces for end-to-end learning under the limited few-shot training data. Sequences are encoded with a temporal block encoder based on Simple Spectral Graph Convolution, a lightweight linear Graph Neural Network backbone. We also include a setting with a transformer. Finally, we propose a similarity-based loss which encourages the alignment of sequences of the same class while preventing the alignment of unrelated sequences. We show state-of-the-art results on NTU-60, NTU-120, Kinetics-skeleton and UWA3D Multiview Activity II.