ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition
This addresses the challenge of explainability in video AI for users needing to understand model decisions, though it is incremental as it builds on existing attention-based methods.
The paper tackled the problem of interpreting deep neural network decisions in video recognition by proposing ST-ABN, a visual explanation method that considers both spatial and temporal information, and demonstrated improved recognition performance on Something-Something datasets V1 and V2.
It is difficult for people to interpret the decision-making in the inference process of deep neural networks. Visual explanation is one method for interpreting the decision-making of deep learning. It analyzes the decision-making of 2D CNNs by visualizing an attention map that highlights discriminative regions. Visual explanation for interpreting the decision-making process in video recognition is more difficult because it is necessary to consider not only spatial but also temporal information, which is different from the case of still images. In this paper, we propose a visual explanation method called spatio-temporal attention branch network (ST-ABN) for video recognition. It enables visual explanation for both spatial and temporal information. ST-ABN acquires the importance of spatial and temporal information during network inference and applies it to recognition processing to improve recognition performance and visual explainability. Experimental results with Something-Something datasets V1 \& V2 demonstrated that ST-ABN enables visual explanation that takes into account spatial and temporal information simultaneously and improves recognition performance.