Enhancing Transformer-based models for Long Sequence Time Series Forecasting via Structured Matrix
This work addresses efficiency bottlenecks in time series forecasting models, though it appears incremental as it optimizes existing Transformer architectures rather than introducing a fundamentally new approach.
The authors tackled the efficiency problem of Transformer-based models for long sequence time series forecasting by proposing a novel architectural framework that replaces self-attention and feed-forward layers with Surrogate Attention Blocks and Surrogate Feed-Forward Neural Network Blocks, achieving an average performance improvement of 12.4% and 61.3% reduction in parameter counts across 10 models and five tasks.
Recently, Transformer-based models for long sequence time series forecasting have demonstrated promising results. The self-attention mechanism as the core component of these Transformer-based models exhibits great potential in capturing various dependencies among data points. Despite these advancements, it has been a subject of concern to improve the efficiency of the self-attention mechanism. Unfortunately, current specific optimization methods are facing the challenges in applicability and scalability for the future design of long sequence time series forecasting models. Hence, in this article, we propose a novel architectural framework that enhances Transformer-based models through the integration of Surrogate Attention Blocks (SAB) and Surrogate Feed-Forward Neural Network Blocks (SFB). The framework reduces both time and space complexity by the replacement of the self-attention and feed-forward layers with SAB and SFB while maintaining their expressive power and architectural advantages. The equivalence of this substitution is fully demonstrated. The extensive experiments on 10 Transformer-based models across five distinct time series tasks demonstrate an average performance improvement of 12.4%, alongside 61.3% reduction in parameter counts.