InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
This addresses a performance bottleneck for developers and users of augmented LLMs, such as those integrated with tools or agents, by reducing resource waste and increasing throughput, though it is incremental as it optimizes an existing paradigm.
The paper tackles the inefficiency in LLM inference systems when handling external interactions, which cause 37-40% recomputation overhead, by introducing InferCept, a framework that minimizes GPU waste and improves serving throughput by 1.6x-2x and request completion by 2x per second.
Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.