Explaining Neural Networks by Decoding Layer Activations
This work addresses the interpretability challenge in neural networks for researchers and practitioners, but it is incremental as it builds on existing autoencoder methods.
The paper tackles the problem of interpreting neural network layers by introducing ClaDec, a classifier-decoder architecture that transforms layer activations into human-understandable representations, showing that it captures more classification-relevant information than conventional autoencoders in image classification tasks.
We present a `CLAssifier-DECoder' architecture (\emph{ClaDec}) which facilitates the comprehension of the output of an arbitrary layer in a neural network (NN). It uses a decoder to transform the non-interpretable representation of the given layer to a representation that is more similar to the domain a human is familiar with. In an image recognition problem, one can recognize what information is represented by a layer by contrasting reconstructed images of \emph{ClaDec} with those of a conventional auto-encoder(AE) serving as reference. We also extend \emph{ClaDec} to allow the trade-off between human interpretability and fidelity. We evaluate our approach for image classification using Convolutional NNs. We show that reconstructed visualizations using encodings from a classifier capture more relevant information for classification than conventional AEs. Relevant code is available at \url{https://github.com/JohnTailor/ClaDec}