RITA: Group Attention is All You Need for Timeseries Analytics
This addresses the problem of inefficient timeseries analytics for applications requiring long sequences, offering a scalable solution with significant performance improvements.
The authors tackled the scalability issue of Transformers for long timeseries by developing RITA, which uses a novel group attention mechanism to reduce time and space complexity while providing theoretical guarantees, resulting in up to 63X speedups and outperforming state-of-the-art methods in accuracy.
Timeseries analytics is of great importance in many real-world applications. Recently, the Transformer model, popular in natural language processing, has been leveraged to learn high quality feature embeddings from timeseries, core to the performance of various timeseries analytics tasks. However, the quadratic time and space complexities limit Transformers' scalability, especially for long timeseries. To address these issues, we develop a timeseries analytics tool, RITA, which uses a novel attention mechanism, named group attention, to address this scalability issue. Group attention dynamically clusters the objects based on their similarity into a small number of groups and approximately computes the attention at the coarse group granularity. It thus significantly reduces the time and space complexity, yet provides a theoretical guarantee on the quality of the computed attention. The dynamic scheduler of RITA continuously adapts the number of groups and the batch size in the training process, ensuring group attention always uses the fewest groups needed to meet the approximation quality requirement. Extensive experiments on various timeseries datasets and analytics tasks demonstrate that RITA outperforms the state-of-the-art in accuracy and is significantly faster -- with speedups of up to 63X.