UnitNorm: Rethinking Normalization for Transformers in Time Series
This work addresses normalization challenges for researchers and practitioners using Transformers in time series tasks, offering a novel method with strong performance gains.
The paper tackled the problem of normalization issues like token shift and sparse attention in Transformers for time series analysis by proposing UnitNorm, which scales input vectors by their norms and modulates attention patterns, resulting in up to a 1.46 decrease in MSE for forecasting and a 4.89% increase in accuracy for classification.
Normalization techniques are crucial for enhancing Transformer models' performance and stability in time series analysis tasks, yet traditional methods like batch and layer normalization often lead to issues such as token shift, attention shift, and sparse attention. We propose UnitNorm, a novel approach that scales input vectors by their norms and modulates attention patterns, effectively circumventing these challenges. Grounded in existing normalization frameworks, UnitNorm's effectiveness is demonstrated across diverse time series analysis tasks, including forecasting, classification, and anomaly detection, via a rigorous evaluation on 6 state-of-the-art models and 10 datasets. Notably, UnitNorm shows superior performance, especially in scenarios requiring robust attention mechanisms and contextual comprehension, evidenced by significant improvements by up to a 1.46 decrease in MSE for forecasting, and a 4.89% increase in accuracy for classification. This work not only calls for a reevaluation of normalization strategies in time series Transformers but also sets a new direction for enhancing model performance and stability. The source code is available at https://anonymous.4open.science/r/UnitNorm-5B84.