Interpretable Spatio-temporal Attention for Video Action Recognition
This work addresses the problem of interpretability in video action recognition for researchers and practitioners, offering a plug-in mechanism that is incremental but improves upon existing methods.
The paper tackles video action recognition by proposing an interpretable spatio-temporal attention mechanism that improves accuracy and localizes discriminative regions and frames without requiring bounding box or temporal labels. Experimental results show superior or comparable accuracy to state-of-the-art methods while enhancing model interpretability.
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose an interpretable and easy plug-in spatial-temporal attention mechanism for video action recognition. For spatial attention, we learn a saliency mask to allow the model to focus on the most salient parts of the feature maps. For temporal attention, we employ a convolutional LSTM based attention mechanism to identify the most relevant frames from an input video. Further, we propose a set of regularizers to ensure that our attention mechanism attends to coherent regions in space and time. Our model not only improves video action recognition accuracy, but also localizes discriminative regions both spatially and temporally, despite being trained in a weakly-supervised manner with only classification labels (no bounding box labels or time frame temporal labels). We evaluate our approach on several public video action recognition datasets with ablation studies. Furthermore, we quantitatively and qualitatively evaluate our model's ability to localize discriminative regions spatially and critical frames temporally. Experimental results demonstrate the efficacy of our approach, showing superior or comparable accuracy with the state-of-the-art methods while increasing model interpretability.