Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systems
This provides a flexible, non-parametric alternative for output prediction in dynamical systems, impacting fields like engineering and science, though it is incremental as it adapts existing transformer methods to a new application.
The paper tackles the problem of predicting dynamical system outputs from noisy observations by showing that transformers in an in-context learning setting can implicitly infer hidden states without test-time updates or explicit model knowledge. In linear-Gaussian regimes, predictions match the Kalman filter, and in nonlinear regimes, performance approaches that of Extended Kalman and particle filters, with robust degradation when parameters are withheld.
Predicting the behavior of a dynamical system from noisy observations of its past outputs is a classical problem encountered across engineering and science. For linear systems with Gaussian inputs, the Kalman filter -- the best linear minimum mean-square error estimator of the state trajectory -- is optimal in the Bayesian sense. For nonlinear systems, Bayesian filtering is typically approached using suboptimal heuristics such as the Extended Kalman Filter (EKF), or numerical methods such as particle filtering (PF). In this work, we show that transformers, employed in an in-context learning (ICL) setting, can implicitly infer hidden states in order to predict the outputs of a wide family of dynamical systems, without test-time gradient updates or explicit knowledge of the system model. Specifically, when provided with a short context of past input-output pairs and, optionally, system parameters, a frozen transformer accurately predicts the current output. In linear-Gaussian regimes, its predictions closely match those of the Kalman filter; in nonlinear regimes, its performance approaches that of EKF and PF. Moreover, prediction accuracy degrades gracefully when key parameters, such as the state-transition matrix, are withheld from the context, demonstrating robustness and implicit parameter inference. These findings suggest that transformer in-context learning provides a flexible, non-parametric alternative for output prediction in dynamical systems, grounded in implicit latent-state estimation.