Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
This provides concrete guidance for designing tokenizers and leveraging transfer learning in time series modeling, particularly for multi-modal forecasting, though it is incremental as it builds on existing tokenization and pretraining methods.
The paper systematically studied how tokenizer design (scaling and quantization) and pretraining affect time series forecasting models, finding that well-designed tokenizers combined with pretraining yield better performance, especially with small vocabularies, while misaligned tokenization can negate pretraining benefits.
Tokenization and transfer learning are two critical components in building state of the art time series foundation models for forecasting. In this work, we systematically study the effect of tokenizer design, specifically scaling and quantization strategies, on model performance, alongside the impact of pretraining versus random initialization. We show that tokenizer configuration primarily governs the representational capacity and stability of the model, while transfer learning influences optimization efficiency and alignment. Using a combination of empirical training experiments and theoretical analyses, we demonstrate that pretrained models consistently leverage well-designed tokenizers more effectively, particularly at smaller vocabulary sizes. Conversely, misaligned tokenization can diminish or even invert the benefits of pretraining. These findings highlight the importance of careful tokenization in time series modeling and suggest that combining small, efficient vocabularies with pretrained weights is especially advantageous in multi-modal forecasting settings, where the overall vocabulary must be shared across modalities. Our results provide concrete guidance for designing tokenizers and leveraging transfer learning in discrete representation learning for continuous signals.