Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent
For researchers in bilevel optimization, two-time-scale stochastic approximation, and adversarial training, this work provides a sharp theoretical characterization of transient dynamics in coupled gradient descent, addressing a critical gap in understanding non-asymptotic behavior.
This paper develops a pseudospectral theory for block-triangular Jacobians in coupled gradient descent, proving that the Kreiss constant is bounded by a function of the coupling norm and spectral radius, and establishing matching lower bounds. The theory yields a finite-horizon iteration-complexity bound of O(K(J)^2 log(1/δ)) for stochastic coupled descent, revealing transient amplification invisible to spectral-radius analysis.
Coupled gradient descent--where the update of one parameter block depends on another--underlies bilevel optimization, two-time-scale stochastic approximation, and adversarial training. When the coupled Jacobian is block-triangular, asymptotic stability is governed by the spectral radii of the diagonal blocks, yet transient amplification before convergence can be arbitrarily large due to non-normality. We develop a sharp pseudospectral theory for such block-triangular Jacobians, proving that the Kreiss constant satisfies $K(J) \leq 2/(1-γ) + \|C\|/(4(1-γ))$ when the diagonal blocks are symmetric with spectral radii at most $γ< 1$, and we establish matching minimax lower bounds. We characterize the critical coupling threshold for spectral instability and extend the analysis to nearly self-referential systems via a Neumann-series perturbation framework. As a consequence, we obtain a finite-horizon iteration-complexity bound of $O(K(J)^2 \log(1/δ))$ for stochastic coupled descent. Framed as scaling laws for non-stationary two-time-scale optimization, our results expose a non-asymptotic, instance-dependent regime of high-dimensional learning dynamics that is invisible to spectral-radius analysis. Experiments on linear-quadratic problems, IQC-based comparisons, and neural-network training confirm the theory.