Inference Graphs for CNN Interpretation
This addresses the interpretability issue for CNN users, but it is incremental as it builds on existing probabilistic modeling techniques.
The authors tackled the problem of interpreting opaque convolutional neural networks (CNNs) by modeling hidden layer activity with probabilistic models, resulting in inference graphs that help understand general inference processes and explain specific image decisions.
Convolutional neural networks (CNNs) have achieved superior accuracy in many visual related tasks. However, the inference process through intermediate layers is opaque, making it difficult to interpret such networks or develop trust in their operation. We propose to model the network hidden layers activity using probabilistic models. The activity patterns in layers of interest are modeled as Gaussian mixture models, and transition probabilities between clusters in consecutive modeled layers are estimated. Based on maximum-likelihood considerations, nodes and paths relevant for network prediction are chosen, connected, and visualized as an inference graph. We show that such graphs are useful for understanding the general inference process of a class, as well as explaining decisions the network makes regarding specific images.