Singular Value Representation: A New Graph Perspective On Neural Networks
This provides a new graph-based perspective for analyzing neural networks, which is incremental but offers specific insights for machine learning researchers.
The paper tackles the problem of understanding neural network internal states by introducing the Singular Value Representation (SVR), a method using SVD factorization to represent weights as a weighted graph of spectral neurons, and demonstrates its utility by revealing a dominant connection in VGG networks and a sparsification effect induced by batch normalization.
We introduce the Singular Value Representation (SVR), a new method to represent the internal state of neural networks using SVD factorization of the weights. This construction yields a new weighted graph connecting what we call spectral neurons, that correspond to specific activation patterns of classical neurons. We derive a precise statistical framework to discriminate meaningful connections between spectral neurons for fully connected and convolutional layers. To demonstrate the usefulness of our approach for machine learning research, we highlight two discoveries we made using the SVR. First, we highlight the emergence of a dominant connection in VGG networks that spans multiple deep layers. Second, we witness, without relying on any input data, that batch normalization can induce significant connections between near-kernels of deep layers, leading to a remarkable spontaneous sparsification phenomenon.