An Efficient Spatio-Temporal Pyramid Transformer for Action Detection
This addresses the problem of high computational cost for action detection in videos for researchers and practitioners, representing an incremental efficiency improvement over existing Transformer methods.
The paper tackles the computational inefficiency of vision Transformers for action detection in long videos by proposing a hierarchical Spatio-Temporal Pyramid Transformer (STPT) that uses local window attention in early stages and global attention in later stages. The result is a 53.6% mAP on THUMOS14 with RGB input alone, outperforming I3D+AFSD by over 10% and matching state-of-the-art AFSD with flow features while using 31% fewer GFLOPs.
The task of action detection aims at deducing both the action category and localization of the start and end moment for each action instance in a long, untrimmed video. While vision Transformers have driven the recent advances in video understanding, it is non-trivial to design an efficient architecture for action detection due to the prohibitively expensive self-attentions over a long sequence of video clips. To this end, we present an efficient hierarchical Spatio-Temporal Pyramid Transformer (STPT) for action detection, building upon the fact that the early self-attention layers in Transformers still focus on local patterns. Specifically, we propose to use local window attention to encode rich local spatio-temporal representations in the early stages while applying global attention modules to capture long-term space-time dependencies in the later stages. In this way, our STPT can encode both locality and dependency with largely reduced redundancy, delivering a promising trade-off between accuracy and efficiency. For example, with only RGB input, the proposed STPT achieves 53.6% mAP on THUMOS14, surpassing I3D+AFSD RGB model by over 10% and performing favorably against state-of-the-art AFSD that uses additional flow features with 31% fewer GFLOPs, which serves as an effective and efficient end-to-end Transformer-based framework for action detection.