Breaking the Ice: Analyzing Cold Start Latency in vLLM
For practitioners deploying vLLM at scale, this work provides a systematic understanding and predictive model of cold start latency, enabling better resource planning.
This paper presents the first detailed performance characterization of vLLM startup latency, breaking it into six steps and showing it is CPU-bound with consistent scaling trends. It develops a lightweight analytical model that accurately predicts startup latency for given hardware, providing actionable guidance for resource planning.
As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study of its startup latency. With major architectural innovations such as the V1 API and the introduction of torch.compile, this paper presents the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate that it is predominantly CPU bound. Each step exhibits consistent and interpretable scaling trends with respect to model-level and system-level parameters, enabling fine-grained attribution of latency sources. Building on these insights, we develop a lightweight analytical model that accurately predicts vLLM startup latency for a given hardware configuration, providing actionable guidance for resource planning in large-scale inference environments. All benchmarking datasets, analysis tools, and prediction scripts are open sourced at https://github.com/upb-cn/vllm-startup-profiler.