MICA: Multivariate Infini Compressive Attention for Time Series Forecasting
This addresses a core bottleneck in multivariate forecasting for applications requiring efficient handling of high-dimensional data, representing a strong incremental improvement.
The paper tackles the scalability challenge of modeling cross-channel dependencies in multivariate time series forecasting with Transformers by proposing MICA, which adds a cross-channel attention mechanism that scales linearly with channel count and context length, reducing forecast error by 5.4% on average and up to 25.4% on individual datasets.
Multivariate forecasting with Transformers faces a core scalability challenge: modeling cross-channel dependencies via attention compounds attention's quadratic sequence complexity with quadratic channel scaling, making full cross-channel attention impractical for high-dimensional time series. We propose Multivariate Infini Compressive Attention (MICA), an architectural design to extend channel-independent Transformers to channel-dependent forecasting. By adapting efficient attention techniques from the sequence dimension to the channel dimension, MICA adds a cross-channel attention mechanism to channel-independent backbones that scales linearly with channel count and context length. We evaluate channel-independent Transformer architectures with and without MICA across multiple forecasting benchmarks. MICA reduces forecast error over its channel-independent counterparts by 5.4% on average and up to 25.4% on individual datasets, highlighting the importance of explicit cross-channel modeling. Moreover, models with MICA rank first among deep multivariate Transformer and MLP baselines. MICA models also scale more efficiently with respect to both channel count and context length than Transformer baselines that compute attention across both the temporal and channel dimensions, establishing compressive attention as a practical solution for scalable multivariate forecasting.