MVAFormer: RGB-based Multi-View Spatio-Temporal Action Recognition with Transformer
This work addresses the challenge of recognizing sequential actions per person in multi-view settings, which is important for applications like surveillance and robotics, but it is incremental as it adapts existing transformer ideas to a specific task.
The paper tackles the problem of multi-view spatio-temporal action recognition (STAR) by proposing MVAFormer, a transformer-based method that uses feature maps to preserve spatial information and divides self-attention for effective view cooperation, achieving a 4.4-point improvement in F-measure over baselines on a new dataset.
Multi-view action recognition aims to recognize human actions using multiple camera views and deals with occlusion caused by obstacles or crowds. In this task, cooperation among views, which generates a joint representation by combining multiple views, is vital. Previous studies have explored promising cooperation methods for improving performance. However, since their methods focus only on the task setting of recognizing a single action from an entire video, they are not applicable to the recently popular spatio-temporal action recognition~(STAR) setting, in which each person's action is recognized sequentially. To address this problem, this paper proposes a multi-view action recognition method for the STAR setting, called MVAFormer. In MVAFormer, we introduce a novel transformer-based cooperation module among views. In contrast to previous studies, which utilize embedding vectors with lost spatial information, our module utilizes the feature map for effective cooperation in the STAR setting, which preserves the spatial information. Furthermore, in our module, we divide the self-attention for the same and different views to model the relationship between multiple views effectively. The results of experiments using a newly collected dataset demonstrate that MVAFormer outperforms the comparison baselines by approximately $4.4$ points on the F-measure.