Non-Stationarity in the Embedding Space of Time Series Foundation Models
For practitioners using TSFMs as feature extractors, this work clarifies that embedding spaces may obscure non-stationarity, which is critical for quality monitoring and change-point analysis.
The paper studies how different forms of non-stationarity (mean shifts, variance changes, linear trends, and persistence) manifest in the embedding spaces of time series foundation models, finding that detectability degrades smoothly and models exhibit distinct failure modes.
Time series foundation models (TSFMs) are widely used as generic feature extractors, yet the notion of non-stationarity in their embedding spaces remains poorly understood. Recent work often conflates non-stationarity with distribution shift, blurring distinctions fundamental to classical time-series analysis and long-standing methodologies such as statistical process control (SPC). In SPC, non-stationarity signals a process leaving a stable regime - via shifts in mean, variance, or emerging trends - and detecting such departures is central to quality monitoring and change-point analysis. Motivated by this diagnostic tradition, we study how different forms of distributional non-stationarity - mean shifts, variance changes, and linear trends - become linearly accessible in TSFM embedding spaces under controlled conditions. We further examine temporal non-stationarity arising from persistence, which reflects violations of weak stationarity due to long-memory or near-unit-root behavior rather than explicit distributional shifts. By sweeping shift strength and probing multiple TSFMs, we find that embedding-space detectability of non-stationarity degrades smoothly and that different models exhibit distinct, model-specific failure modes.