IFNSO: Iteration-Free Newton-Schulz Orthogonalization
This work addresses efficiency issues in orthogonalization for optimizers like Muon and Stiefel manifold optimization, representing an incremental improvement over existing methods.
The paper tackles the computational overhead of the Newton-Schulz iteration for orthogonalization by proposing IFNSO, which consolidates it into an iteration-free formulation with learnable coefficients, achieving superior performance in experiments.
The Newton-Schulz (NS) iteration has become a key technique for orthogonalization in optimizers such as Muon and for optimization on the Stiefel manifold. Despite its effectiveness, the conventional NS iteration incurs significant computational overhead due to repeated high-dimensional matrix multiplications. To overcome these limitations, we propose Iteration-Free Newton-Schulz Orthogonalization (IFNSO), a novel framework that consolidates the traditional iterative structure into a unified and Iteration-Free formulation. By analyzing the contribution of individual matrix powers, we streamline the process by removing insignificant terms and introducing a polynomial with learnable coefficients. These coefficients are optimized to ensure both superior computational efficiency and stable convergence. Extensive experiments demonstrate that IFNSO achieves superior performance compared to existing methods. Our code is available at: https://github.com/greekinRoma/Unified_Newton_Schulz_Orthogonalization.