OccamVTS: Distilling Vision Models to 1% Parameters for Time Series Forecasting
This work addresses inefficiencies in applying vision models to time series forecasting, offering a lightweight solution for domains like finance or IoT, though it is incremental as it builds on existing distillation and vision model techniques.
The paper tackled the problem of using large vision models for time series forecasting by revealing that 99% of their parameters are unnecessary, and proposed OccamVTS, a distillation framework that reduces parameters to 1% while improving accuracy, achieving state-of-the-art performance across benchmarks.
Time series forecasting is fundamental to diverse applications, with recent approaches leverage large vision models (LVMs) to capture temporal patterns through visual representations. We reveal that while vision models enhance forecasting performance, 99% of their parameters are unnecessary for time series tasks. Through cross-modal analysis, we find that time series align with low-level textural features but not high-level semantics, which can impair forecasting accuracy. We propose OccamVTS, a knowledge distillation framework that extracts only the essential 1% of predictive information from LVMs into lightweight networks. Using pre-trained LVMs as privileged teachers, OccamVTS employs pyramid-style feature alignment combined with correlation and feature distillation to transfer beneficial patterns while filtering out semantic noise. Counterintuitively, this aggressive parameter reduction improves accuracy by eliminating overfitting to irrelevant visual features while preserving essential temporal patterns. Extensive experiments across multiple benchmark datasets demonstrate that OccamVTS consistently achieves state-of-the-art performance with only 1% of the original parameters, particularly excelling in few-shot and zero-shot scenarios.