United We Pretrain, Divided We Fail! Representation Learning for Time Series by Pretraining on 75 Datasets at Once
This addresses the challenge of effective representation learning for time series data, which is incremental as it adapts pretraining methods from NLP and vision to time series.
The paper tackles the problem of pretraining for time series representation learning by introducing a self-supervised contrastive approach that learns from 75 diverse datasets, disproving the belief that multi-dataset pretraining doesn't work for time series, and it outperforms supervised and other self-supervised methods in low-data regimes.
In natural language processing and vision, pretraining is utilized to learn effective representations. Unfortunately, the success of pretraining does not easily carry over to time series due to potential mismatch between sources and target. Actually, common belief is that multi-dataset pretraining does not work for time series! Au contraire, we introduce a new self-supervised contrastive pretraining approach to learn one encoding from many unlabeled and diverse time series datasets, so that the single learned representation can then be reused in several target domains for, say, classification. Specifically, we propose the XD-MixUp interpolation method and the Soft Interpolation Contextual Contrasting (SICC) loss. Empirically, this outperforms both supervised training and other self-supervised pretraining methods when finetuning on low-data regimes. This disproves the common belief: We can actually learn from multiple time series datasets, even from 75 at once.