Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit
This work provides theoretical insights into implicit regularization in overparametrized models, which is an incremental advancement for machine learning researchers.
The authors tackled the problem of understanding implicit regularization in deep linear networks by extending a geometric framework to infinite depth and linking Riemannian geometry to training dynamics for matrix completion, with rigorous analysis and numerics.
The deep linear network (DLN) is a model for implicit regularization in gradient based optimization of overparametrized learning architectures. Training the DLN corresponds to a Riemannian gradient flow, where the Riemannian metric is defined by the architecture of the network and the loss function is defined by the learning task. We extend this geometric framework, obtaining explicit expressions for the volume form, including the case when the network has infinite depth. We investigate the link between the Riemannian geometry and the training asymptotics for matrix completion with rigorous analysis and numerics. We propose that implicit regularization is a result of bias towards high state space volume.