Towards Long-Context Time Series Foundation Models
This addresses a practical problem in domains like healthcare where long, multivariate time series are common, representing an incremental improvement over existing methods.
The paper tackled the limitation of existing time series foundation models in handling long, multivariate data by introducing a compressive memory mechanism, enabling encoder-only models to effectively model intra-variate dependencies and demonstrating benefits with the MOMENT family of models.
Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. However, most of these models are designed to handle short univariate time series as an input. This limits their practical use, especially in domains such as healthcare with copious amounts of long and multivariate data with strong temporal and intra-variate dependencies. Our study bridges this gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies. We demonstrate the benefits of our approach by imbuing MOMENT, a recent family of multi-task time series foundation models, with the multivariate context.