Unlocking the Power of Mixture-of-Experts for Task-Aware Time Series Analytics
This work addresses the challenge of making MoE architectures task-aware for time series applications, which is incremental as it builds on existing MoE methods by introducing task-specific adaptations.
The paper tackled the problem of adapting Mixture-of-Experts (MoE) architectures for time series analytics by addressing task-agnostic routing and channel correlation modeling, resulting in a novel framework called PatchMoE that achieved state-of-the-art performance across five downstream tasks.
Time Series Analysis is widely used in various real-world applications such as weather forecasting, financial fraud detection, imputation for missing data in IoT systems, and classification for action recognization. Mixture-of-Experts (MoE), as a powerful architecture, though demonstrating effectiveness in NLP, still falls short in adapting to versatile tasks in time series analytics due to its task-agnostic router and the lack of capability in modeling channel correlations. In this study, we propose a novel, general MoE-based time series framework called PatchMoE to support the intricate ``knowledge'' utilization for distinct tasks, thus task-aware. Based on the observation that hierarchical representations often vary across tasks, e.g., forecasting vs. classification, we propose a Recurrent Noisy Gating to utilize the hierarchical information in routing, thus obtaining task-sepcific capability. And the routing strategy is operated on time series tokens in both temporal and channel dimensions, and encouraged by a meticulously designed Temporal \& Channel Load Balancing Loss to model the intricate temporal and channel correlations. Comprehensive experiments on five downstream tasks demonstrate the state-of-the-art performance of PatchMoE.