Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
This work provides theoretical insights into optimization for deep linear networks, which is incremental but addresses a known bottleneck in non-convex analysis.
The paper tackles the challenge of analyzing the loss functions of deep linear networks by studying the local geometric landscape around critical points, deriving an error bound that quantifies distance to the critical set in terms of gradient norm, and demonstrating linear convergence of gradient descent in numerical experiments.
The optimization foundations of deep linear networks have recently received significant attention. However, due to their inherent non-convexity and hierarchical structure, analyzing the loss functions of deep linear networks remains a challenging task. In this work, we study the local geometric landscape of the regularized squared loss of deep linear networks around each critical point. Specifically, we derive a closed-form characterization of the critical point set and establish an error bound for the regularized loss under mild conditions on network width and regularization parameters. Notably, this error bound quantifies the distance from a point to the critical point set in terms of the current gradient norm, which can be used to derive linear convergence of first-order methods. To support our theoretical findings, we conduct numerical experiments and demonstrate that gradient descent converges linearly to a critical point when optimizing the regularized loss of deep linear networks.