Continual Transformers: Redundancy-Free Attention for Online Inference
This addresses the inefficiency of Transformers for real-time applications like video or audio processing, offering a significant speedup without performance loss.
The paper tackles the problem of Transformers being inefficient for online inference on time-series data due to redundancy from overlapping token sequences, proposing a novel attention formulation that enables token-by-token processing and reduces floating point operations by up to 63x while maintaining predictive performance.
Transformers in their common form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference on time-series data entails considerable redundancy due to the overlap in successive token sequences. In this work, we propose novel formulations of the Scaled Dot-Product Attention, which enable Transformers to perform efficient online token-by-token inference on a continual input stream. Importantly, our modifications are purely to the order of computations, while the outputs and learned weights are identical to those of the original Transformer Encoder. We validate our Continual Transformer Encoder with experiments on the THUMOS14, TVSeries and GTZAN datasets with remarkable results: Our Continual one- and two-block architectures reduce the floating point operations per prediction by up to 63x and 2.6x, respectively, while retaining predictive performance.