A Probabilistic Basis for Low-Rank Matrix Learning
This work provides a theoretical foundation for low-rank matrix learning methods, benefiting researchers and practitioners in machine learning and statistics by making Bayesian inference more tractable and efficient.
The paper addresses the lack of understanding of probability distributions underlying nuclear norm penalties in low-rank matrix learning by analytically characterizing the distribution with density proportional to e^{-λ||X||_*}. The result is an improved MCMC algorithm that eliminates hyperparameter tuning and enhances accuracy and efficiency in matrix denoising and completion tasks.
Low rank inference on matrices is widely conducted by optimizing a cost function augmented with a penalty proportional to the nuclear norm $\Vert \cdot \Vert_*$. However, despite the assortment of computational methods for such problems, there is a surprising lack of understanding of the underlying probability distributions being referred to. In this article, we study the distribution with density $f(X)\propto e^{-λ\Vert X\Vert_*}$, finding many of its fundamental attributes to be analytically tractable via differential geometry. We use these facts to design an improved MCMC algorithm for low rank Bayesian inference as well as to learn the penalty parameter $λ$, obviating the need for hyperparameter tuning when this is difficult or impossible. Finally, we deploy these to improve the accuracy and efficiency of low rank Bayesian matrix denoising and completion algorithms in numerical experiments.