Deep Double Descent for Time Series Forecasting: Avoiding Undertrained Models
This addresses the problem of undertrained models in time series forecasting for researchers and practitioners, offering an incremental improvement by optimizing training rather than architecture.
The paper investigates training schemas for deep learning models in time series forecasting, showing that deep double descent occurs and overfitting can be reversed with more epochs, leading to state-of-the-art results in nearly 70% of 72 benchmarks.
Deep learning models, particularly Transformers, have achieved impressive results in various domains, including time series forecasting. While existing time series literature primarily focuses on model architecture modifications and data augmentation techniques, this paper explores the training schema of deep learning models for time series; how models are trained regardless of their architecture. We perform extensive experiments to investigate the occurrence of deep double descent in several Transformer models trained on public time series data sets. We demonstrate epoch-wise deep double descent and that overfitting can be reverted using more epochs. Leveraging these findings, we achieve state-of-the-art results for long sequence time series forecasting in nearly 70% of the 72 benchmarks tested. This suggests that many models in the literature may possess untapped potential. Additionally, we introduce a taxonomy for classifying training schema modifications, covering data augmentation, model inputs, model targets, time series per model, and computational budget.