Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting
This work addresses the problem of efficient training for time series forecasting in data-scarce scenarios, offering incremental insights into modality-agnostic model properties.
The paper investigates the transfer of pre-trained language models to time series forecasting in low-data settings, finding that design choices like upstream post-training and tokenizers significantly impact validation loss, with a persistent transfer gap where language models outperform randomly initialized models even after convergence.
Recent works have demonstrated the effectiveness of adapting pre-trained language models (LMs) for forecasting time series in the low-data regime. We build upon these findings by analyzing the effective transfer from language models to time series forecasting under various design choices including upstream post-training, time series tokenizer and language backbone size. In the low-data regime, these design choices have a significant impact on the validation loss, with clear-cut choices that outperform others. Contrary to Hernandez et al. (2021), we observe that the validation loss of the LMs continues to smoothly decrease long after the validation loss of the randomly initialized models has converged, leading to a non-vanishing transfer gap that holds across design choices. These findings not only help shed light on the effective use of compute-efficient training for time series, but also open the way for the study of modality-agnostic properties of data distributions leveraged by these models.