Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks
This work addresses a theoretical gap for researchers in deep learning, offering insights into how orthonormality affects convergence, though it is incremental as it builds on existing practical uses of orthonormality.
The paper tackles the lack of theoretical analysis for orthonormality in neural networks by providing convergence analysis for training orthonormal deep linear neural networks, showing that Riemannian gradient descent with appropriate initialization converges at a linear rate for a class of loss functions.
Enforcing orthonormal or isometric property for the weight matrices has been shown to enhance the training of deep neural networks by mitigating gradient exploding/vanishing and increasing the robustness of the learned networks. However, despite its practical performance, the theoretical analysis of orthonormality in neural networks is still lacking; for example, how orthonormality affects the convergence of the training process. In this letter, we aim to bridge this gap by providing convergence analysis for training orthonormal deep linear neural networks. Specifically, we show that Riemannian gradient descent with an appropriate initialization converges at a linear rate for training orthonormal deep linear neural networks with a class of loss functions. Unlike existing works that enforce orthonormal weight matrices for all the layers, our approach excludes this requirement for one layer, which is crucial to establish the convergence guarantee. Our results shed light on how increasing the number of hidden layers can impact the convergence speed. Experimental results validate our theoretical analysis.