SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition
This work addresses memory constraints in skeleton-based action recognition for applications like video analysis, though it is incremental as it builds on transformer-based approaches with a novel partitioning strategy.
The paper tackled the problem of skeleton-based action recognition by addressing the memory inefficiency of transformer-based methods that capture all joint-frame correlations, proposing SkateFormer which partitions joints and frames into four skeletal-temporal relation types for selective attention, resulting in outperforming recent state-of-the-art methods on benchmark datasets.
Skeleton-based action recognition, which classifies human actions based on the coordinates of joints and their connectivity within skeleton data, is widely utilized in various scenarios. While Graph Convolutional Networks (GCNs) have been proposed for skeleton data represented as graphs, they suffer from limited receptive fields constrained by joint connectivity. To address this limitation, recent advancements have introduced transformer-based methods. However, capturing correlations between all joints in all frames requires substantial memory resources. To alleviate this, we propose a novel approach called Skeletal-Temporal Transformer (SkateFormer) that partitions joints and frames based on different types of skeletal-temporal relation (Skate-Type) and performs skeletal-temporal self-attention (Skate-MSA) within each partition. We categorize the key skeletal-temporal relations for action recognition into a total of four distinct types. These types combine (i) two skeletal relation types based on physically neighboring and distant joints, and (ii) two temporal relation types based on neighboring and distant frames. Through this partition-specific attention strategy, our SkateFormer can selectively focus on key joints and frames crucial for action recognition in an action-adaptive manner with efficient computation. Extensive experiments on various benchmark datasets validate that our SkateFormer outperforms recent state-of-the-art methods.