U-shaped Transformer: Retain High Frequency Context in Time Series Analysis
This work addresses a specific bottleneck in time series analysis for industrial applications, offering an incremental improvement by hybridizing transformer and MLP advantages.
The paper tackles the problem of transformers losing high-frequency context in time series prediction by proposing a U-shaped Transformer that incorporates skip-layer connections from Unet to retain this information, achieving advanced performance across multiple datasets with relatively low cost.
Time series prediction plays a crucial role in various industrial fields. In recent years, neural networks with a transformer backbone have achieved remarkable success in many domains, including computer vision and NLP. In time series analysis domain, some studies have suggested that even the simplest MLP networks outperform advanced transformer-based networks on time series forecast tasks. However, we believe these findings indicate there to be low-rank properties in time series sequences. In this paper, we consider the low-pass characteristics of transformers and try to incorporate the advantages of MLP. We adopt skip-layer connections inspired by Unet into traditional transformer backbone, thus preserving high-frequency context from input to output, namely U-shaped Transformer. We introduce patch merge and split operation to extract features with different scales and use larger datasets to fully make use of the transformer backbone. Our experiments demonstrate that the model performs at an advanced level across multiple datasets with relatively low cost.