Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping
This addresses optimization stability and accuracy issues in deep learning, particularly for adversarial scenarios, though it appears incremental as it builds on spectral regularization methods.
The paper tackles the problem of high layer condition numbers in deep networks, which degrade optimization and accuracy, by introducing Chebyshev Moment Regularization (CMR) to directly optimize layer spectra, resulting in a reduction of mean layer condition numbers by approximately 10^3 and restoring test accuracy from about 10% to 86% in an adversarial setting.
We introduce \textbf{Chebyshev Moment Regularization (CMR)}, a simple, architecture-agnostic loss that directly optimizes layer spectra. CMR jointly controls spectral edges via a log-condition proxy and shapes the interior via Chebyshev moments, with a decoupled, capped mixing rule that preserves task gradients. We prove strictly monotone descent for the condition proxy, bounded moment gradients, and orthogonal invariance. In an adversarial ``$κ$-stress'' setting (MNIST, 15-layer MLP), \emph{compared to vanilla training}, CMR reduces mean layer condition numbers by $\sim\!10^3$ (from $\approx3.9\!\times\!10^3$ to $\approx3.4$ in 5 epochs), increases average gradient magnitude, and restores test accuracy ( $\approx10\%\!\to\!\approx86\%$ ). These results support \textbf{optimization-driven spectral preconditioning}: directly steering models toward well-conditioned regimes for stable, accurate learning.