Self-Distilled Representation Learning for Time Series
This work addresses the need for effective self-supervised learning in time-series analysis, offering a novel approach that avoids modality-specific biases, though it is incremental as it adapts an existing framework to a new domain.
The authors tackled the problem of self-supervised learning for time-series data by proposing a non-contrastive, self-distillation method based on a student-teacher scheme that predicts latent representations from masked views, achieving competitive results in classification and forecasting tasks on datasets like UCR, UEA, ETT, and Electricity.
Self-supervised learning for time-series data holds potential similar to that recently unleashed in Natural Language Processing and Computer Vision. While most existing works in this area focus on contrastive learning, we propose a conceptually simple yet powerful non-contrastive approach, based on the data2vec self-distillation framework. The core of our method is a student-teacher scheme that predicts the latent representation of an input time series from masked views of the same time series. This strategy avoids strong modality-specific assumptions and biases typically introduced by the design of contrastive sample pairs. We demonstrate the competitiveness of our approach for classification and forecasting as downstream tasks, comparing with state-of-the-art self-supervised learning methods on the UCR and UEA archives as well as the ETT and Electricity datasets.