Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
This addresses the adaptation challenge for TSFMs in forecasting tasks, offering a practical solution for researchers and practitioners, though it is incremental as it builds on existing pruning and fine-tuning techniques.
The paper tackles the problem of Time Series Foundation Models (TSFMs) underperforming smaller specialized models after fine-tuning by proposing a structured pruning method to regularize fine-tuning, resulting in significant performance improvements and often achieving state-of-the-art results on benchmarks.
Scaling laws motivate the development of Time Series Foundation Models (TSFMs) that pre-train vast parameters and achieve remarkable zero-shot forecasting performance. Surprisingly, even after fine-tuning, TSFMs cannot consistently outperform smaller, specialized models trained on full-shot downstream data. A key question is how to realize effective adaptation of TSFMs for a target forecasting task. Through empirical studies on various TSFMs, the pre-trained models often exhibit inherent sparsity and redundancy in computation, suggesting that TSFMs have learned to activate task-relevant network substructures to accommodate diverse forecasting tasks. To preserve this valuable prior knowledge, we propose a structured pruning method to regularize the subsequent fine-tuning process by focusing it on a more relevant and compact parameter space. Extensive experiments on seven TSFMs and six benchmarks demonstrate that fine-tuning a smaller, pruned TSFM significantly improves forecasting performance compared to fine-tuning original models. This prune-then-finetune paradigm often enables TSFMs to achieve state-of-the-art performance and surpass strong specialized baselines. Source code is made publicly available at https://github.com/SJTU-DMTai/Prune-then-Finetune.