Two-Stream temporal transformer for video action classification
This work addresses video understanding for applications like action recognition and robotics, but it is incremental as it builds on existing transformer methods.
The paper tackles video action classification by introducing a two-stream transformer that extracts spatio-temporal information from content and optical flow, achieving excellent classification results on three well-known human activity datasets.
Motion representation plays an important role in video understanding and has many applications including action recognition, robot and autonomous guidance or others. Lately, transformer networks, through their self-attention mechanism capabilities, have proved their efficiency in many applications. In this study, we introduce a new two-stream transformer video classifier, which extracts spatio-temporal information from content and optical flow representing movement information. The proposed model identifies self-attention features across the joint optical flow and temporal frame domain and represents their relationships within the transformer encoder mechanism. The experimental results show that our proposed methodology provides excellent classification results on three well-known video datasets of human activities.