Stiffness: A New Perspective on Generalization in Neural Networks
This provides a new perspective on generalization for neural network researchers, though it is incremental as it builds on existing concepts without solving a major bottleneck.
The authors introduced 'stiffness' as a measure of how a gradient step on one example affects loss on another, linking it to generalization in neural networks, and demonstrated its dependence on factors like class membership and learning rate across datasets including CIFAR-100, where stiffness revealed super-class awareness.
In this paper we develop a new perspective on generalization of neural networks by proposing and investigating the concept of a neural network stiffness. We measure how stiff a network is by looking at how a small gradient step in the network's parameters on one example affects the loss on another example. Higher stiffness suggests that a network is learning features that generalize. In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, 3) training iteration, and 4) learning rate. We present experiments on MNIST, FASHION MNIST, and CIFAR-10/100 using fully-connected and convolutional neural networks, as well as on a transformer-based NLP model. We demonstrate the connection between stiffness and generalization, and observe its dependence on learning rate. When training on CIFAR-100, the stiffness matrix exhibits a coarse-grained behavior indicative of the model's awareness of super-class membership. In addition, we measure how stiffness between two data points depends on their mutual input-space distance, and establish the concept of a dynamical critical length -- a distance below which a parameter update based on a data point influences its neighbors.