Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking
This addresses the challenge for service providers in efficiently managing LLM workloads with strict latency requirements and resource constraints, representing an incremental improvement over existing methods.
The paper tackles the problem of interference in shared clusters running multiple LLM services, proposing OmniServe, a system that uses CPU-GPU attention piggybacking to improve SLO attainment for latency-sensitive services by up to 1.48x and best-effort service throughput by up to 9.85x compared to state-of-the-art systems.
Nowadays, service providers often deploy multiple types of LLM services within shared clusters. While the service colocation improves resource utilization, it introduces significant interference risks for latency-sensitive (LS) services-which have strict SLO requirements for inference latency-and severely constrain the service capacity of best-effort (BE) services due to limited available memory. To address interference, existing systems typically rely on reserving headroom to constrain BE resource usage. However, this approach's coarse granularity compromises the SLO compliance of the latency-sensitive service and unnecessarily restricts the generation potential of the best effort service. In this paper, we propose OmniServe, a novel LLM serving system that efficiently harnesses both CPU and GPU resources to mitigate interference and improve throughput. Central to OmniServe is the Attention Piggybacking mechanism, which effectively offloads the Attention computation of BE services to CPUs on the fly. This mechanism also facilitates asynchronous communication between CPU and GPU streams, preventing GPUs from being blocked while aggregating Attention results. Additionally, OmniServe incorporates a dynamic batching control policy to adapt to fluctuating request arrivals, facilitating Dense module computation using layer-wise batching. Experimental results show that OmniServe improves the SLO attainment rate for LS services by up to $1.48\times$ while enhancing BE serving throughput by up to $9.85\times$ compared to state-of-the-art systems.