Enhancing Channel-Independent Time Series Forecasting via Cross-Variate Patch Embedding
This work addresses a specific bottleneck in time series forecasting for researchers and practitioners by improving channel-independent models with a novel cross-variate module, representing an incremental advancement.
The paper tackles the problem of channel-independent time series forecasting models omitting cross-variate dependencies by proposing Cross-Variate Patch Embeddings (CVPE), a lightweight module that injects cross-variate context into patch embeddings, resulting in enhanced performance of the Time-LLM model across seven real-world datasets.
Transformers have recently gained popularity in time series forecasting due to their ability to capture long-term dependencies. However, many existing models focus only on capturing temporal dependencies while omitting intricate relationships between variables. Recent models have tried tackling this by explicitly modeling both cross-time and cross-variate dependencies through a sequential or unified attention mechanism, but they are entirely channel dependent (CD) across all layers, making them potentially susceptible to overfitting. To address this, we propose Cross-Variate Patch Embeddings (CVPE), a lightweight CD module that injects cross-variate context into channel-independent (CI) models by simply modifying the patch embedding process. We achieve this by adding a learnable positional encoding and a lightweight router-attention block to the vanilla patch embedding layer. We then integrate CVPE into Time-LLM, a multimodal CI forecasting model, to demonstrate its effectiveness in capturing cross-variate dependencies and enhance the CI model's performance. Extensive experimental results on seven real-world datasets show that our enhanced Time-LLM outperforms the original baseline model simply by incorporating the CVPE module, with no other changes.