A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorization
This provides a new theoretical approach to analyze implicit regularization in deep learning, though it appears incremental as it builds on existing discrete gradient dynamics methods.
The authors tackled the problem of explaining implicit regularization in deep matrix factorization by analyzing saddle point escaping stages, proving that for rank-R matrix reconstruction, the model converges to a second-order critical point after R stages, which was experimentally verified.
Implicit regularization is an important way to interpret neural networks. Recent theory starts to explain implicit regularization with the model of deep matrix factorization (DMF) and analyze the trajectory of discrete gradient dynamics in the optimization process. These discrete gradient dynamics are relatively small but not infinitesimal, thus fitting well with the practical implementation of neural networks. Currently, discrete gradient dynamics analysis has been successfully applied to shallow networks but encounters the difficulty of complex computation for deep networks. In this work, we introduce another discrete gradient dynamics approach to explain implicit regularization, i.e. landscape analysis. It mainly focuses on gradient regions, such as saddle points and local minima. We theoretically establish the connection between saddle point escaping (SPE) stages and the matrix rank in DMF. We prove that, for a rank-R matrix reconstruction, DMF will converge to a second-order critical point after R stages of SPE. This conclusion is further experimentally verified on a low-rank matrix reconstruction problem. This work provides a new theory to analyze implicit regularization in deep learning.