Domain Adaptation for Time series Transformers using One-step fine-tuning
This addresses data shift and generalization issues in time series Transformers for domains like energy and environmental monitoring, though it appears incremental.
The paper tackles the problem of time series prediction for domains with limited data by introducing a one-step fine-tuning approach that adds source domain data to target domains and uses gradual unfreezing, improving over state-of-the-art baselines by 4.35% and 11.54% on indoor temperature and wind power datasets.
The recent breakthrough of Transformers in deep learning has drawn significant attention of the time series community due to their ability to capture long-range dependencies. However, like other deep learning models, Transformers face limitations in time series prediction, including insufficient temporal understanding, generalization challenges, and data shift issues for the domains with limited data. Additionally, addressing the issue of catastrophic forgetting, where models forget previously learned information when exposed to new data, is another critical aspect that requires attention in enhancing the robustness of Transformers for time series tasks. To address these limitations, in this paper, we pre-train the time series Transformer model on a source domain with sufficient data and fine-tune it on the target domain with limited data. We introduce the \emph{One-step fine-tuning} approach, adding some percentage of source domain data to the target domains, providing the model with diverse time series instances. We then fine-tune the pre-trained model using a gradual unfreezing technique. This helps enhance the model's performance in time series prediction for domains with limited data. Extensive experimental results on two real-world datasets show that our approach improves over the state-of-the-art baselines by 4.35% and 11.54% for indoor temperature and wind power prediction, respectively.