RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
This addresses resource efficiency and latency challenges in LLM inference serving systems, particularly in resource-constrained environments, though it appears incremental by combining and optimizing existing techniques.
The paper tackles the trade-offs in LLM inference serving between hybrid batching and disaggregated serving by proposing RAPID-Serve, which concurrently executes prefill and decode on the same GPU(s) to meet latency SLOs while improving throughput, achieving up to 4.1x unconstrained and 32x SLO-constrained throughput gains.
Two widely adopted techniques for LLM inference serving systems today are hybrid batching and disaggregated serving. A hybrid batch combines prefill and decode tokens of different requests in the same batch to improve resource utilization and throughput at the cost of increased latency per token. In contrast, disaggregated serving decouples compute-bound prefill and bandwidth-bound decode phases to optimize for service level objectives (SLOs) at the cost of resource under-utilization and KV-cache transfer overheads. To address the limitations of these techniques, we propose RAPID-Serve: a technique to concurrently execute prefill and decode on the same GPU(s) to meet latency SLOs while maintaining high throughput and efficient resource utilization. Furthermore, we propose Adaptive Resource Management for runtime compute resource allocation, optionally leveraging CU masking (a fine-grained Compute Unit partitioning feature on AMD Instinct\textsuperscript{TM} GPUs). RAPID-Serve provides up to 4.1x (average 1.7x) unconstrained throughput improvement and 32x and higher (average 4.9x) throughput improvement under SLO constraints, showing it as an effective strategy compared to the state-of-the-art approaches, particularly in resource-constrained environments.