Enhancing Time Series Forecasting via Logic-Inspired Regularization
This work addresses a domain-specific issue in time series forecasting by introducing a novel regularization technique to enhance Transformer models, though it appears incremental as it builds on existing methods.
The paper tackles the problem of inefficient token dependencies in Transformer-based time series forecasting by proposing Attention Logic Regularization (Attn-L-Reg), a plug-and-play method that sparsifies attention maps to use fewer but more effective dependencies, resulting in improved prediction performance as confirmed by experiments and theoretical analysis.
Time series forecasting (TSF) plays a crucial role in many applications. Transformer-based methods are one of the mainstream techniques for TSF. Existing methods treat all token dependencies equally. However, we find that the effectiveness of token dependencies varies across different forecasting scenarios, and existing methods ignore these differences, which affects their performance. This raises two issues: (1) What are effective token dependencies? (2) How can we learn effective dependencies? From a logical perspective, we align Transformer-based TSF methods with the logical framework and define effective token dependencies as those that ensure the tokens as atomic formulas (Issue 1). We then align the learning process of Transformer methods with the process of obtaining atomic formulas in logic, which inspires us to design a method for learning these effective dependencies (Issue 2). Specifically, we propose Attention Logic Regularization (Attn-L-Reg), a plug-and-play method that guides the model to use fewer but more effective dependencies by making the attention map sparse, thereby ensuring the tokens as atomic formulas and improving prediction performance. Extensive experiments and theoretical analysis confirm the effectiveness of Attn-L-Reg.