In-Context Learning Under Regime Change
For practitioners deploying foundation models in non-stationary environments, this work provides theoretical guarantees and practical evidence that transformers can detect and adapt to regime shifts, with demonstrated improvements in disease and financial forecasting.
The paper formalizes in-context change-point detection for transformers, proving existence of models that adapt to regime shifts, and shows that trained transformers match optimal baselines on synthetic tasks while improving real-world forecasting for infectious diseases and financial volatility without retraining.
Non-stationary sequences arise naturally in control, forecasting, and decision-making. The data-generating process shifts at unknown times, and models must detect the change, discard or downweight obsolete evidence, and adapt to new dynamics on the fly. Transformer-based foundation models increasingly rely on in-context learning for time series forecasting, tabular prediction, and continuous control. As these models are deployed in non-stationary environments, understanding their ability to detect and adapt to regime shifts is important. We formalize this as an in-context change-point detection problem and formally establish the existence of transformer models that solve this problem. Our construction demonstrates that model complexity, in layers and parameters, depends on the level of information available about the change-point location, from no knowledge to knowing exact timing. We validate our results with experiments on synthetic linear regression and linear dynamical systems, where trained transformers match the performance of optimal baselines across information levels. We also show that encoding and incorporating changepoint knowledge indeed improves the real-world performance of a pretrained foundation models on infectious disease forecasting and on financial volatility forecasting around Federal Open Market Committee (FOMC) announcements without retraining, demonstrating practical applicability to real-world regime changes.