A Supervised Contrastive Learning Pretrain-Finetune Approach for Time Series
This work addresses a domain-specific problem for time series machine learning researchers, offering an incremental improvement in representation extraction and knowledge transfer.
The paper tackles the challenge of transferring knowledge from pretraining to finetuning datasets in time series foundation models by introducing a supervised contrastive learning pretraining procedure that uses a probabilistic similarity metric to guide fine-tuning, resulting in promising experimental efficacy.
Foundation models have recently gained attention within the field of machine learning thanks to its efficiency in broad data processing. While researchers had attempted to extend this success to time series models, the main challenge is effectively extracting representations and transferring knowledge from pretraining datasets to the target finetuning dataset. To tackle this issue, we introduce a novel pretraining procedure that leverages supervised contrastive learning to distinguish features within each pretraining dataset. This pretraining phase enables a probabilistic similarity metric, which assesses the likelihood of a univariate sample being closely related to one of the pretraining datasets. Subsequently, using this similarity metric as a guide, we propose a fine-tuning procedure designed to enhance the accurate prediction of the target data by aligning it more closely with the learned dynamics of the pretraining datasets. Our experiments have shown promising results which demonstrate the efficacy of our approach.