Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models
This addresses the problem of adapting large models to specialized, data-scarce domains like healthcare, though it is incremental as it builds on existing PEFT techniques.
The paper tackled the challenge of fine-tuning Time Series Foundation Models for sensitive healthcare tasks like ICU vitals forecasting for sepsis patients, achieving state-of-the-art results with FourierFT fine-tuning only 2,400 parameters compared to 700K in benchmarks.
Time Series Foundation Models (TSFMs) have recently garnered attention for their ability to model complex, large-scale time series data across domains such as retail, finance, and transportation. However, their application to sensitive, domain-specific fields like healthcare remains challenging, primarily due to the difficulty of fine-tuning these models for specialized, out-of-domain tasks with scarce publicly available datasets. In this work, we explore the use of Parameter-Efficient Fine-Tuning (PEFT) techniques to address these limitations, focusing on healthcare applications, particularly ICU vitals forecasting for sepsis patients. We introduce and evaluate two selective (BitFit and LayerNorm Tuning) and two additive (VeRA and FourierFT) PEFT techniques on multiple configurations of the Chronos TSFM for forecasting vital signs of sepsis patients. Our comparative analysis demonstrates that some of these PEFT methods outperform LoRA in terms of parameter efficiency and domain adaptation, establishing state-of-the-art (SOTA) results in ICU vital forecasting tasks. Interestingly, FourierFT applied to the Chronos (Tiny) variant surpasses the SOTA model while fine-tuning only 2,400 parameters compared to the 700K parameters of the benchmark.