Understanding Convolutional Networks with APPLE : Automatic Patch Pattern Labeling for Explanation
This work addresses the need for interpretability in deep learning for researchers and practitioners, but it is incremental as it builds on existing explanation methods.
The paper tackles the problem of explaining how convolutional networks make classifications by developing an algorithm to identify important neurons and automatically label input image patches that activate them, demonstrating its use for gaining insight into network decision-making.
With the success of deep learning, recent efforts have been focused on analyzing how learned networks make their classifications. We are interested in analyzing the network output based on the network structure and information flow through the network layers. We contribute an algorithm for 1) analyzing a deep network to find neurons that are 'important' in terms of the network classification outcome, and 2)automatically labeling the patches of the input image that activate these important neurons. We propose several measures of importance for neurons and demonstrate that our technique can be used to gain insight into, and explain how a network decomposes an image to make its final classification.