Spatio-Temporal Perturbations for Video Attribution
This work addresses the problem of interpreting opaque neural networks for video inputs, which is incremental as it builds on existing perturbation-based methods but adapts them for video-specific structures.
The paper tackles the challenge of explaining video understanding networks by proposing a generic perturbation-based attribution method with a novel regularization term for spatiotemporal smoothness, and introduces objective metrics for evaluation, showing effectiveness through both subjective and objective comparisons.
The attribution method provides a direction for interpreting opaque neural networks in a visual way by identifying and visualizing the input regions/pixels that dominate the output of a network. Regarding the attribution method for visually explaining video understanding networks, it is challenging because of the unique spatiotemporal dependencies existing in video inputs and the special 3D convolutional or recurrent structures of video understanding networks. However, most existing attribution methods focus on explaining networks taking a single image as input and a few works specifically devised for video attribution come short of dealing with diversified structures of video understanding networks. In this paper, we investigate a generic perturbation-based attribution method that is compatible with diversified video understanding networks. Besides, we propose a novel regularization term to enhance the method by constraining the smoothness of its attribution results in both spatial and temporal dimensions. In order to assess the effectiveness of different video attribution methods without relying on manual judgement, we introduce reliable objective metrics which are checked by a newly proposed reliability measurement. We verified the effectiveness of our method by both subjective and objective evaluation and comparison with multiple significant attribution methods.