Neural Activation Patterns (NAPs): Visual Explainability of Learned Concepts
This provides a tool for researchers and practitioners to better interpret neural network layers, though it is incremental as it builds on activation analysis methods.
The paper tackles the problem of limited layer-level concept discovery in neural networks by introducing Neural Activation Patterns (NAPs), a method that analyzes entire activation distributions to group inputs with similar activation profiles, enabling visualization and interpretation of learned concepts, and it was tested on various networks to complement existing techniques.
A key to deciphering the inner workings of neural networks is understanding what a model has learned. Promising methods for discovering learned features are based on analyzing activation values, whereby current techniques focus on analyzing high activation values to reveal interesting features on a neuron level. However, analyzing high activation values limits layer-level concept discovery. We present a method that instead takes into account the entire activation distribution. By extracting similar activation profiles within the high-dimensional activation space of a neural network layer, we find groups of inputs that are treated similarly. These input groups represent neural activation patterns (NAPs) and can be used to visualize and interpret learned layer concepts. We release a framework with which NAPs can be extracted from pre-trained models and provide a visual introspection tool that can be used to analyze NAPs. We tested our method with a variety of networks and show how it complements existing methods for analyzing neural network activation values.