Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach
This work addresses the challenge of training robust time series foundation models in heterogeneous environments, which is crucial for applications across diverse domains like finance, healthcare, and IoT, though it appears incremental as it builds on existing federated learning and representation learning techniques.
The paper tackles the problem of training time series foundation models (TSFMs) in heterogeneous environments where temporal dynamics vary across domains, which causes gradient conflicts and degrades representation quality in existing mixed-batch approaches. The proposed federated learning method addresses bi-level heterogeneity by enforcing domain-invariant representations and domain-aware aggregation, resulting in TSFMs that consistently outperform centralized and federated baselines in forecasting tasks and achieve competitive zero-shot performance at scale.
Heterogeneity in time series data is more pronounced than in vision or language, as temporal dynamics vary substantially across domains and tasks. Existing efforts on training time series foundation models (TSFMs) from scratch are often trained with mixed-batch strategies that merge large-scale datasets, which can cause gradient conflicts and degrade representation quality. To address this, we propose a fine-grained learning method that distills invariant knowledge from heterogeneous series while reducing cross-domain interference. We characterize heterogeneity at two levels: inter-domain and intra-domain. To tackle this bi-level heterogeneity, we design a federated learning method that mitigates intra-domain conflicts by enforcing domain-invariant and semantically consistent representations through local regularization, and addresses inter-domain discrepancies by enhancing cross-domain collaboration via domain-aware aggregation. Experiments across diverse benchmarks show that TSFMs trained with our method consistently outperform both centralized and federated TSFM baselines in point and probabilistic forecasting, while also achieving competitive zero-shot performance at scale, offering a flexible pathway for training TSFMs from scratch in heterogeneous environments.