General Time-series Model for Universal Knowledge Representation of Multivariate Time-Series data
This addresses a central problem in multivariate time series analysis for applications requiring robust foundation models, though it appears incremental by building on existing attention and pre-training concepts.
The paper tackles the problem of universal knowledge representation for multivariate time series foundation models by proposing a General Time-series Model (GTM) that learns time granularity-aware representations and uses an autoregressive blank infilling pre-training framework, achieving state-of-the-art performance across generative tasks like long-term forecasting, anomaly detection, and imputation.
Universal knowledge representation is a central problem for multivariate time series(MTS) foundation models and yet remains open. This paper investigates this problem from the first principle and it makes four folds of contributions. First, a new empirical finding is revealed: time series with different time granularities (or corresponding frequency resolutions) exhibit distinct joint distributions in the frequency domain. This implies a crucial aspect of learning universal knowledge, one that has been overlooked by previous studies. Second, a novel Fourier knowledge attention mechanism is proposed to enable learning time granularity-aware representations from both the temporal and frequency domains. Third, an autoregressive blank infilling pre-training framework is incorporated to time series analysis for the first time, leading to a generative tasks agnostic pre-training strategy. To this end, we develop the General Time-series Model (GTM), a unified MTS foundation model that addresses the limitation of contemporary time series models, which often require token, pre-training, or model-level customizations for downstream tasks adaption. Fourth, extensive experiments show that GTM outperforms state-of-the-art (SOTA) methods across all generative tasks, including long-term forecasting, anomaly detection, and imputation.