Learning Novel Transformer Architecture for Time-series Forecasting
This work addresses a domain-specific problem for time-series forecasting researchers and practitioners, offering incremental improvements through architecture search.
The authors tackled limitations in Transformer-based models for time-series forecasting by proposing AutoFormer-TS, a framework that uses a differentiable neural architecture search method to explore alternative architectures, resulting in superior forecasting accuracy across benchmarks.
Despite the success of Transformer-based models in the time-series prediction (TSP) tasks, the existing Transformer architecture still face limitations and the literature lacks comprehensive explorations into alternative architectures. To address these challenges, we propose AutoFormer-TS, a novel framework that leverages a comprehensive search space for Transformer architectures tailored to TSP tasks. Our framework introduces a differentiable neural architecture search (DNAS) method, AB-DARTS, which improves upon existing DNAS approaches by enhancing the identification of optimal operations within the architecture. AutoFormer-TS systematically explores alternative attention mechanisms, activation functions, and encoding operations, moving beyond the traditional Transformer design. Extensive experiments demonstrate that AutoFormer-TS consistently outperforms state-of-the-art baselines across various TSP benchmarks, achieving superior forecasting accuracy while maintaining reasonable training efficiency.