AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
This addresses the computational bottleneck in transformer inference for applications requiring fast processing, though it is incremental as it builds on existing memoization ideas with system-level optimizations.
The paper tackles the problem of long inference time in transformer models by using memoization to accelerate the self-attention mechanism, achieving an average 22% reduction in inference latency (up to 68%) with negligible accuracy loss.
Transformer models gain popularity because of their superior inference accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing works on transformer inference acceleration have limitations caused by either the modification of transformer architectures or the need of specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the self-attention mechanism in transformers without the above limitations. Built upon a unique observation that there is rich similarity in attention computation across inference sequences, we build a memoization database that leverages the emerging big memory system. We introduce a novel embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 22% inference-latency reduction on average (up to 68%) with negligible loss in inference accuracy.