TopoAct: Visually Exploring the Shape of Activations in Deep Learning
This work addresses the interpretability challenge in deep learning for researchers and practitioners, offering a novel visualization tool for network analysis and diagnosis, though it is incremental in extending existing topological methods to activation spaces.
The researchers tackled the problem of understanding how deep neural networks achieve high performance by developing TopoAct, a visual exploration system that uses topological data analysis to study the shape and organization of neuron activations across layers, providing insights into learned representations.
Deep neural networks such as GoogLeNet, ResNet, and BERT have achieved impressive performance in tasks such as image and text classification. To understand how such performance is achieved, we probe a trained deep neural network by studying neuron activations, i.e., combinations of neuron firings, at various layers of the network in response to a particular input. With a large number of inputs, we aim to obtain a global view of what neurons detect by studying their activations. In particular, we develop visualizations that show the shape of the activation space, the organizational principle behind neuron activations, and the relationships of these activations within a layer. Applying tools from topological data analysis, we present TopoAct, a visual exploration system to study topological summaries of activation vectors. We present exploration scenarios using TopoAct that provide valuable insights into learned representations of neural networks. We expect TopoAct to give a topological perspective that enriches the current toolbox of neural network analysis, and to provide a basis for network architecture diagnosis and data anomaly detection.