DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams
This addresses the need for low-latency inference on resource-constrained devices for applications like audio, video, and text streams, representing an incremental improvement over previous continual transformers.
The paper tackled the problem of high computational redundancy in Transformer models for real-time inference on data streams by proposing DeepCoT, which achieved comparable performance to non-continual baselines while reducing running time by up to two orders of magnitude.
Transformer-based models have dramatically increased their size and parameter count to tackle increasingly complex tasks. At the same time, there is a growing demand for low-latency inference on resource-constrained devices that achieves high performance. In particular, stream data inference is typically performed over a sliding temporal window, leading to highly redundant computations. The recent Continual Transformers have addressed this issue, but they can only be effectively used in shallow models, which limits their scope and generalization power. In this paper, we propose the Deep Continual Transformer (DeepCoT), a redundancy-free encoder-only model that can be applied over existing deep encoder architectures with minimal changes. In our experiments over audio, video, and text streams, we show that DeepCoTs retain comparative performance to their non-continual baselines while offering a linear computational cost for all Transformer layers, which reduces up to two orders of magnitude in the running time compared to previous efficient models.