Fine-Tuning a Time Series Foundation Model with Wasserstein Loss
This work addresses a specific limitation in time series forecasting models for researchers and practitioners, but it is incremental as it modifies an existing approach rather than introducing a new paradigm.
The paper tackles the problem of using cross-entropy loss for time series forecasting with LLM-based foundation models, which is suboptimal because it ignores distances between classes, and proposes using Wasserstein loss instead, resulting in significant improvements in point estimation on 22 zero-shot datasets.
Inspired by recent advancements in large language models (LLMs) for Natural Language Processing (NLP), there has been a surge in research focused on developing foundational models for time series forecasting. One approach involves training LLM architectures on tokenized time series data using cross-entropy loss. Although this method has demonstrated promising results, cross-entropy loss is primarily designed for classification tasks and does not account for the distance between classes. To address this limitation, we propose using the Wasserstein loss for such architectures. To validate our approach, we fine-tuned a foundational time series model on $22$ zero-shot datasets, comparing the performance of cross-entropy loss with that of Wasserstein loss. Our results demonstrate that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.