Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting
For practitioners using TSFMs in safety-critical domains like traffic forecasting, this work exposes hidden failures in aggregate metrics and provides a simple fix, though the problem and solution are domain-specific.
Standard benchmarks for time series foundation models (TSFMs) hide severe failures during regime transitions (e.g., free-flow to congested traffic). Stratified evaluation reveals that transition-regime MAE reaches 11 mph vs. 3 mph overall, and 90% prediction interval coverage drops to 55%. A proposed post-hoc method (BMA) improves transition coverage while preserving overall accuracy.
Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes. We introduce regime-stratified evaluation and apply it to three TSFMs on two standard traffic speed benchmarks. Traffic exhibits abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. When we stratify by traffic regime, both accuracy and prediction-interval coverage degrade sharply during transitions: transition-regime MAE reaches 11 mph (versus 3 mph overall), and empirical coverage of 90% prediction intervals drops as low as 55%. These failures are invisible in aggregate metrics because free-flow observations dominate the sample. A simple historical conditional baseline (sampling from per-sensor training distributions) achieves better transition coverage than any TSFM, but has far worse overall accuracy. We propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline's transition coverage while preserving the TSFM's accuracy. Our results suggest that TSFM benchmarks should incorporate regime-aware evaluation to surface failures that aggregate metrics hide.