NeuralDivergence: Exploring and Understanding Neural Networks by Comparing Activation Distributions
This addresses the need for interpretability in high-stakes deep learning applications, though it is incremental as it builds on existing visualization tools by extending them to multi-instance and cross-layer comparisons.
The authors tackled the problem of understanding neural network decision mechanisms by developing NeuralDivergence, an interactive visualization system that compares activation distributions across layers, classes, and instances, enabling users to gain better insights into model behavior.
As deep neural networks are increasingly used in solving high-stake problems, there is a pressing need to understand their internal decision mechanisms. Visualization has helped address this problem by assisting with interpreting complex deep neural networks. However, current tools often support only single data instances, or visualize layers in isolation. We present NeuralDivergence, an interactive visualization system that uses activation distributions as a high-level summary of what a model has learned. NeuralDivergence enables users to interactively summarize and compare activation distributions across layers, classes, and instances (e.g., pairs of adversarial attacked and benign images), helping them gain better understanding of neural network models.