SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
This work addresses the implicit regularization effects in neural network training for researchers, though it is incremental as it builds on prior studies of SGD biases.
The paper investigates how Stochastic Gradient Descent (SGD) with weight decay inherently biases neural networks toward learning low-rank weight matrices, showing this effect intensifies with smaller batch sizes, higher learning rates, or stronger weight decay, and finds it has a marginal impact on test performance.
We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.