Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
This addresses the problem of computational inefficiency in video action recognition models for researchers and practitioners, though it is incremental as it builds on existing attention methods.
The paper tackles the challenge of efficiently modeling attention for video action recognition by proposing a What-Where-When (W3) module that jointly learns what, where, and when to focus on, achieving new state-of-the-art performance on multiple benchmarks.
Attentive video modeling is essential for action recognition in unconstrained videos due to their rich yet redundant information over space and time. However, introducing attention in a deep neural network for action recognition is challenging for two reasons. First, an effective attention module needs to learn what (objects and their local motion patterns), where (spatially), and when (temporally) to focus on. Second, a video attention module must be efficient because existing action recognition models already suffer from high computational cost. To address both challenges, a novel What-Where-When (W3) video attention module is proposed. Departing from existing alternatives, our W3 module models all three facets of video attention jointly. Crucially, it is extremely efficient by factorizing the high-dimensional video feature data into low-dimensional meaningful spaces (1D channel vector for `what' and 2D spatial tensors for `where'), followed by lightweight temporal attention reasoning. Extensive experiments show that our attention model brings significant improvements to existing action recognition models, achieving new state-of-the-art performance on a number of benchmarks.