BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecasting
This work addresses efficiency and alignment issues in time series forecasting for applications with memory constraints, representing an incremental improvement.
The paper tackles the problem of long-term time series forecasting with transformers by addressing the quadratic complexity of self-attention and misalignment of uniform patching, introducing BSAT and L-RoPE to achieve competitive performance at high compression rates.
Long-term time series forecasting using transformers is hampered by the quadratic complexity of self-attention and the rigidity of uniform patching, which may be misaligned with the data's semantic structure. In this paper, we introduce the \textit{B-Spline Adaptive Tokenizer (BSAT)}, a novel, parameter-free method that adaptively segments a time series by fitting it with B-splines. BSAT algorithmically places tokens in high-curvature regions and represents each variable-length basis function as a fixed-size token, composed of its coefficient and position. Further, we propose a hybrid positional encoding that combines a additive learnable positional encoding with Rotary Positional Embedding featuring a layer-wise learnable base: L-RoPE. This allows each layer to attend to different temporal dependencies. Our experiments on several public benchmarks show that our model is competitive with strong performance at high compression rates. This makes it particularly well-suited for use cases with strong memory constraints.