Two-Stream Transformer Architecture for Long Video Understanding
This addresses data and memory efficiency issues in transformers for long-form video tasks, benefiting domains with computational constraints.
The paper tackles the problem of long video understanding by introducing a two-stream transformer architecture called STAN, which achieves state-of-the-art performance on several tasks and can classify videos up to two minutes in length on a single GPU.
Pure vision transformer architectures are highly effective for short video classification and action recognition tasks. However, due to the quadratic complexity of self attention and lack of inductive bias, transformers are resource intensive and suffer from data inefficiencies. Long form video understanding tasks amplify data and memory efficiency problems in transformers making current approaches unfeasible to implement on data or memory restricted domains. This paper introduces an efficient Spatio-Temporal Attention Network (STAN) which uses a two-stream transformer architecture to model dependencies between static image features and temporal contextual features. Our proposed approach can classify videos up to two minutes in length on a single GPU, is data efficient, and achieves SOTA performance on several long video understanding tasks.