ART: Attention Run-time Termination for Efficient Large Language Model Decoding
For LLM practitioners, ART provides a lightweight, orthogonal mechanism to improve decoding efficiency without sacrificing accuracy.
ART addresses the memory bandwidth bottleneck in long-context LLM decoding by tracking accumulated attention outputs and terminating KV block accesses when contributions become negligible, achieving 20% higher generation throughput at large batch sizes with comparable accuracy.
Long-context decoding in Large Language Models (LLMs) is severely constrained by the memory bandwidth required to fetch the extensive Key-Value (KV) cache. Most existing KV management methods rely on key-only pruning before decoding, despite the evidence that attention outputs depend jointly on keys and values, as incorporating values in their methods incurs prohibitive additional overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. This design makes ART orthogonal to existing key-based KV cache management methods, enabling seamless integration with them. Experiments on LongBench benchmarks show that ART achieves 20% higher generation throughput in large batch size than state-of-the-art baseline while maintaining comparable accuracy.