Two-Layer Linear Auto-Regressive Models Estimate Latent States
This work provides a theoretical understanding of how auto-regressive models learn latent representations in sequential data, which is important for the machine learning community studying representation learning in sequence models.
This paper shows that two-layer linear auto-regressive models trained on data from partially observed linear dynamical systems learn to approximate Kalman filtering, with the learned hidden representation matching the optimal state estimates up to a similarity transformation. The authors provide finite-sample guarantees on prediction error, parameter estimation, and latent state recovery.
Auto-regressive models have emerged as powerful tools for sequential data, from language to video. Understanding how and why these models learn latent representations remains an open theoretical question. In this work, we demonstrate that when trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, we show that the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal (Kalman) filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from three main insights. First, we establish that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error. Second, we show that despite non-convexity, the two-layer optimization landscape is benign, i.e., all stationary points are either strict saddles or global minima. Finally, as our main contributions, we provide finite-sample guarantees on prediction error, parameter estimation error, and latent state recovery. Numerical simulations support the theoretical results and demonstrate that the latent representations of auto-regressive models recover state estimates.