Take an Irregular Route: Enhance the Decoder of Time-Series Forecasting Transformer
This work addresses a specific bottleneck in time-series forecasting for IoT systems by enhancing decoder design, though it is incremental as it builds on existing Transformer paradigms.
The paper tackles the under-explored decoder design in time-series forecasting Transformers by proposing FPPformer, which uses bottom-up and top-down architectures in the encoder and decoder, respectively, to build a full hierarchy, achieving promising performance verified on twelve benchmarks with six state-of-the-art baselines.
With the development of Internet of Things (IoT) systems, precise long-term forecasting method is requisite for decision makers to evaluate current statuses and formulate future policies. Currently, Transformer and MLP are two paradigms for deep time-series forecasting and the former one is more prevailing in virtue of its exquisite attention mechanism and encoder-decoder architecture. However, data scientists seem to be more willing to dive into the research of encoder, leaving decoder unconcerned. Some researchers even adopt linear projections in lieu of the decoder to reduce the complexity. We argue that both extracting the features of input sequence and seeking the relations of input and prediction sequence, which are respective functions of encoder and decoder, are of paramount significance. Motivated from the success of FPN in CV field, we propose FPPformer to utilize bottom-up and top-down architectures respectively in encoder and decoder to build the full and rational hierarchy. The cutting-edge patch-wise attention is exploited and further developed with the combination, whose format is also different in encoder and decoder, of revamped element-wise attention in this work. Extensive experiments with six state-of-the-art baselines on twelve benchmarks verify the promising performances of FPPformer and the importance of elaborately devising decoder in time-series forecasting Transformer. The source code is released in https://github.com/OrigamiSL/FPPformer.