Spectral Statistics of the Sample Covariance Matrix for High Dimensional Linear Gaussians
This addresses the reliability of OLS in high-dimensional linear Gaussian systems, highlighting a critical failure mode in stable dynamics, which is incremental as it builds on existing spectral theory.
The paper tackles the problem of estimating high-dimensional stable state transition matrices using ordinary least squares (OLS) from noisy trajectories, showing that OLS performance heavily depends on the spectral statistics of the sample covariance matrix. It reveals a phase transition where OLS becomes transient and estimation error worsens with more iterations, specifically when the eigenvalue leaves the complex half unit disc, with the largest eigenvalue scaling as Ω(⌊N/n⌋ e^{α_λ n}).
Performance of ordinary least squares(OLS) method for the \emph{estimation of high dimensional stable state transition matrix} $A$(i.e., spectral radius $ρ(A)<1$) from a single noisy observed trajectory of the linear time invariant(LTI)\footnote{Linear Gaussian (LG) in Markov chain literature} system $X_{-}:(x_0,x_1, \ldots,x_{N-1})$ satisfying \begin{equation} x_{t+1}=Ax_{t}+w_{t}, \hspace{10pt} \text{ where } w_{t} \thicksim N(0,I_{n}), \end{equation} heavily rely on negative moments of the sample covariance matrix: $(X_{-}X_{-}^{*})=\sum_{i=0}^{N-1}x_{i}x_{i}^{*}$ and singular values of $EX_{-}^{*}$, where $E$ is a rectangular Gaussian ensemble $E=[w_0, \ldots, w_{N-1}]$. Negative moments requires sharp estimates on all the eigenvalues $λ_{1}\big(X_{-}X_{-}^{*}\big) \geq \ldots \geq λ_{n}\big(X_{-}X_{-}^{*}\big) \geq 0$. Leveraging upon recent results on spectral theorem for non-Hermitian operators in \cite{naeem2023spectral}, along with concentration of measure phenomenon and perturbation theory(Gershgorins' and Cauchys' interlacing theorem) we show that only when $A=A^{*}$, typical order of $λ_{j}\big(X_{-}X_{-}^{*}\big) \in \big[N-n\sqrt{N}, N+n\sqrt{N}\big]$ for all $j \in [n]$. However, in \emph{high dimensions} when $A$ has only one distinct eigenvalue $λ$ with geometric multiplicity of one, then as soon as eigenvalue leaves \emph{complex half unit disc}, largest eigenvalue suffers from curse of dimensionality: $λ_{1}\big(X_{-}X_{-}^{*}\big)=Ω\big( \lfloor\frac{N}{n}\rfloor e^{α_λn} \big)$, while smallest eigenvalue $λ_{n}\big(X_{-}X_{-}^{*}\big) \in (0, N+\sqrt{N}]$. Consequently, OLS estimator incurs a \emph{phase transition} and becomes \emph{transient: increasing iteration only worsens estimation error}, all of this happening when the dynamics are generated from stable systems.