A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization
This work addresses a theoretical gap in deep matrix factorization for researchers in optimization and machine learning, offering incremental insights into loss landscape properties.
The paper tackles the optimization foundations of deep matrix factorization by analyzing its loss landscape, providing a closed-form characterization of critical points and conditions for their types, which explains why gradient-based methods converge to local minimizers.
Despite its wide range of applications across various domains, the optimization foundations of deep matrix factorization (DMF) remain largely open. In this work, we aim to fill this gap by conducting a comprehensive study of the loss landscape of the regularized DMF problem. Toward this goal, we first provide a closed-form characterization of all critical points of the problem. Building on this, we establish precise conditions under which a critical point is a local minimizer, a global minimizer, a strict saddle point, or a non-strict saddle point. Leveraging these results, we derive a necessary and sufficient condition under which every critical point is either a local minimizer or a strict saddle point. This provides insights into why gradient-based methods almost always converge to a local minimizer of the regularized DMF problem. Finally, we conduct numerical experiments to visualize its loss landscape to support our theory.