Implicit Regularization in Matrix Sensing via Mirror Descent
This provides theoretical insight into implicit regularization for matrix recovery, but is incremental as it extends known results to mirror descent.
The paper analyzes mirror descent for matrix sensing, showing it converges to a global minimizer that minimizes quantities related to nuclear norm, Frobenius norm, and von Neumann entropy, and recovers low-rank matrices under assumptions similar to nuclear-norm minimization.
We study discrete-time mirror descent applied to the unregularized empirical risk in matrix sensing. In both the general case of rectangular matrices and the particular case of positive semidefinite matrices, a simple potential-based analysis in terms of the Bregman divergence allows us to establish convergence of mirror descent -- with different choices of the mirror maps -- to a matrix that, among all global minimizers of the empirical risk, minimizes a quantity explicitly related to the nuclear norm, the Frobenius norm, and the von Neumann entropy. In both cases, this characterization implies that mirror descent, a first-order algorithm minimizing the unregularized empirical risk, recovers low-rank matrices under the same set of assumptions that are sufficient to guarantee recovery for nuclear-norm minimization. When the sensing matrices are symmetric and commute, we show that gradient descent with full-rank factorized parametrization is a first-order approximation to mirror descent, in which case we obtain an explicit characterization of the implicit bias of gradient flow as a by-product.