Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading
This addresses cost reduction for cloud-based LLM serving, but it is incremental as it optimizes existing methods rather than introducing a new paradigm.
The paper tackles the high cost of GPU instances for LLM inference in the cloud by proposing InferSave, a VM selection framework with KV cache offloading, which improves cost efficiency by up to 73.7% for online workloads and up to 20.19% for offline workloads.
LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads.