Mutual information estimation for graph convolutional neural networks
This work addresses the need for better interpretability in deep learning, particularly for graph convolutional networks, but it is incremental as it extends existing mutual information analysis to a new architecture type.
The authors tackled the problem of explaining why specific neural network architectures achieve superior predictive accuracy by proposing an architecture-agnostic method for tracking internal representations and creating mutual information planes during training. They demonstrated this method on graph-based neural networks using citation data, comparing how inductive biases affect the mutual information plane relative to fully connected networks.
Measuring model performance is a key issue for deep learning practitioners. However, we often lack the ability to explain why a specific architecture attains superior predictive accuracy for a given data set. Often, validation accuracy is used as a performance heuristic quantifying how well a network generalizes to unseen data, but it does not capture anything about the information flow in the model. Mutual information can be used as a measure of the quality of internal representations in deep learning models, and the information plane may provide insights into whether the model exploits the available information in the data. The information plane has previously been explored for fully connected neural networks and convolutional architectures. We present an architecture-agnostic method for tracking a network's internal representations during training, which are then used to create the mutual information plane. The method is exemplified for graph-based neural networks fitted on citation data. We compare how the inductive bias introduced in graph-based architectures changes the mutual information plane relative to a fully connected neural network.