Baguan-TS: A Sequence-Native In-Context Learning Model for Time Series Forecasting with Covariates
This addresses the need for efficient, gradient-free adaptation in time series forecasting, particularly for applications like energy datasets, though it appears incremental as it builds on existing transformer-based ICL methods.
The paper tackled the problem of in-context learning for time series forecasting with covariates by bridging the gap between raw-sequence representation and inference-time adaptation, resulting in Baguan-TS achieving the highest win rate and significant reductions in point and probabilistic forecasting metrics on benchmarks.
Transformers enable in-context learning (ICL) for rapid, gradient-free adaptation in time series forecasting, yet most ICL-style approaches rely on tabularized, hand-crafted features, while end-to-end sequence models lack inference-time adaptation. We bridge this gap with a unified framework, Baguan-TS, which integrates the raw-sequence representation learning with ICL, instantiated by a 3D Transformer that attends jointly over temporal, variable, and context axes. To make this high-capacity model practical, we tackle two key hurdles: (i) calibration and training stability, improved with a feature-agnostic, target-space retrieval-based local calibration; and (ii) output oversmoothing, mitigated via context-overfitting strategy. On public benchmark with covariates, Baguan-TS consistently outperforms established baselines, achieving the highest win rate and significant reductions in both point and probabilistic forecasting metrics. Further evaluations across diverse real-world energy datasets demonstrate its robustness, yielding substantial improvements.