Joint Embeddings Go Temporal
This work addresses the need for robust representation learning in time series, potentially enabling future foundation models, though it appears incremental as it adapts an existing method to a new domain.
The authors tackled the problem of self-supervised learning for time series by adapting Joint-Embedding Predictive Architectures (JEPA) to create TS-JEPA, which matches or surpasses state-of-the-art baselines on classification and forecasting tasks.
Self-supervised learning has seen great success recently in unsupervised representation learning, enabling breakthroughs in natural language and image processing. However, these methods often rely on autoregressive and masked modeling, which aim to reproduce masked information in the input, which can be vulnerable to the presence of noise or confounding variables. To address this problem, Joint-Embedding Predictive Architectures (JEPA) has been introduced with the aim to perform self-supervised learning in the latent space. To leverage these advancements in the domain of time series, we introduce Time Series JEPA (TS-JEPA), an architecture specifically adapted for time series representation learning. We validate TS-JEPA on both classification and forecasting, showing that it can match or surpass current state-of-the-art baselines on different standard datasets. Notably, our approach demonstrates a strong performance balance across diverse tasks, indicating its potential as a robust foundation for learning general representations. Thus, this work lays the groundwork for developing future time series foundation models based on Joint Embedding.