Interpretation of Deep Temporal Representations by Selective Visualization of Internally Activated Nodes
This work addresses the interpretability challenge for researchers and practitioners using deep learning on temporal data, but it is incremental as it builds on existing visualization and uncertainty methods.
The paper tackles the problem of interpreting temporal deep neural networks by proposing two frameworks for visualizing learned representations, which extract highly activated periods and characterize sub-sequences with clustering and uncertainty calculations.
Recently deep neural networks demonstrate competitive performances in classification and regression tasks for many temporal or sequential data. However, it is still hard to understand the classification mechanisms of temporal deep neural networks. In this paper, we propose two new frameworks to visualize temporal representations learned from deep neural networks. Given input data and output, our algorithm interprets the decision of temporal neural network by extracting highly activated periods and visualizes a sub-sequence of input data which contributes to activate the units. Furthermore, we characterize such sub-sequences with clustering and calculate the uncertainty of the suggested type and actual data. We also suggest Layer-wise Relevance from the output of a unit, not from the final output, with backward Monte-Carlo dropout to show the relevance scores of each input point to activate units with providing a visual representation of the uncertainty about this impact.