Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning
For practitioners training large neural networks, Zeta provides a principled fix to a vulnerability in matrix-aware optimizers like Muon, leading to more reliable and efficient training.
Zeta introduces a dual whitening optimizer that applies coordinate whitening before spectral whitening to fix scale heterogeneity in momentum matrices, improving condition number and reducing orthogonalization error. It matches or surpasses baselines in language modeling (0.6B to 8B parameters), MoE architectures, and vision tasks, achieving faster convergence and better generalization.
Large-scale neural network training increasingly relies on matrix-aware optimizers that exploit the structure of weight parameters beyond element-wise adaptation. However, existing matrix-aware methods such as Muon have an underappreciated vulnerability: their core operation, Newton-Schulz iteration, depends critically on input conditioning, yet the raw momentum matrices exhibit severe coordinate-wise scale heterogeneity. In this paper, we first verify this scale heterogeneity through a chi-square uniformity test, showing that intra-matrix scale imbalance is prevalent across Transformer layers and that coordinate whitening effectively corrects it. Motivated by this finding, we propose Zeta, a dual whitening optimizer that applies coordinate whitening and spectral whitening in a strictly ordered pipeline. The ordering is not a tunable choice but follows from a mathematical dependency: coordinate whitening establishes the statistical isotropy that spectral whitening requires to function reliably. We further prove that this dual pipeline strictly reduces orthogonalization error relative to pure spectral methods by improving the condition number of the input. Empirically, Zeta matches or surpasses strong baselines across language modeling (0.6B to 8B parameters), mixture-of-experts architectures, and vision tasks, demonstrating that resolving scale imbalance before orthogonalization leads to faster convergence and better generalization. Code is available at https://gitcode.com/kevin259/MindSpeed.