On the Benefits of Weight Normalization for Overparameterized Matrix Sensing
This work provides a theoretical understanding of weight normalization for researchers in optimization and deep learning, though it is incremental as it focuses on a specific problem.
The paper tackles the overparameterized matrix sensing problem by applying weight normalization, proving that it achieves linear convergence and yields an exponential speedup over standard methods, with iteration and sample complexity improving polynomially as overparameterization increases.
While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an exponential speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing.