Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
This work addresses time series forecasting challenges in domains like energy and traffic, offering incremental improvements to Transformer efficiency and accuracy.
The paper tackled the problems of locality-agnostic attention and quadratic memory complexity in Transformers for time series forecasting by proposing convolutional self-attention and LogSparse Transformer, achieving improved forecasting accuracy on synthetic and real-world datasets compared to state-of-the-art methods.
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.