BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
This addresses a performance bottleneck in latency-insensitive applications using large models, offering incremental improvements in throughput optimization.
The paper tackles the problem of sub-optimal throughput in offline batch inference for auto-regressive large models due to conflicts between resource overlapping and prefix sharing, presenting BlendServe, which achieves up to 1.44x throughput boost compared to industry standards like vLLM and SGLang.
Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to $1.44\times$ throughput boost compared to widely-used industry standards, vLLM and SGLang.