Low-rank bias, weight decay, and model merging in neural networks
This work provides theoretical insights into regularization effects for researchers in optimization and neural network theory, but it is incremental as it extends known properties to new contexts.
The paper investigates the low-rank structure of weight matrices in neural networks at stationary points with L2 regularization, showing properties like alignment, norm preservation, and low-rank bias, and demonstrates that summing weights of networks trained on orthogonal inputs yields comparable performance on both tasks.
We explore the low-rank structure of the weight matrices in neural networks at the stationary points (limiting solutions of optimization algorithms) with $L2$ regularization (also known as weight decay). We show several properties of such deep neural networks, induced by $L2$ regularization. In particular, for a stationary point we show alignment of the parameters and the gradient, norm preservation across layers, and low-rank bias: properties previously known in the context of solution of gradient descent/flow type algorithms. Experiments show that the assumptions made in the analysis only mildly affect the observations. In addition, we investigate a multitask learning phenomenon enabled by $L2$ regularization and low-rank bias. In particular, we show that if two networks are trained, such that the inputs in the training set of one network are approximately orthogonal to the inputs in the training set of the other network, the new network obtained by simply summing the weights of the two networks will perform as well on both training sets as the respective individual networks. We demonstrate this for shallow ReLU neural networks trained by gradient descent, as well as deep linear networks trained by gradient flow.