Transformers and Their Roles as Time Series Foundation Models
This provides theoretical foundations for transformers in time series analysis, which is incremental but important for applications like forecasting.
The paper analyzes transformers as time series foundation models, demonstrating they can fit autoregressive models via gradient descent and proving MOIRAI can handle arbitrary covariates, with experiments supporting these theoretical findings.
We give a comprehensive analysis of transformers as time series foundation models, focusing on their approximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, a multivariate time series foundation model capable of handling an arbitrary number of covariates. We prove that it is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into its design and empirical success. For generalization, we establish bounds for pretraining when the data satisfies Dobrushin's condition. Experiments support our theoretical findings, highlighting the efficacy of transformers as time series foundation models.