LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
For researchers and engineers developing LLM serving systems, LLM-Emu enables low-cost, realistic online experimentation without expensive GPU hardware.
LLM-Emu is a serving-native emulator for vLLM that replaces GPU forward execution with profile-sampled latency and synthetic tokens, achieving TPOT and ITL within 4.8% absolute error, E2E latency within 5.3%, and output throughput within 1.9% compared to real vLLM serving.
Realistic evaluation of LLM serving systems requires online workloads, dynamic arrivals, queueing, and the serving engine's local scheduling for execution batching, but running such experiments on GPUs is expensive. Existing simulators reduce this cost, but often operate offline or in time-warped mode, re-implement serving-engine schedulers, or require accurate operator/kernel-level latency models. We present LLM-Emu, a serving-native emulator for vLLM that preserves the production HTTP, scheduling, KV-cache, and output-processing paths while replacing only GPU forward execution with profile-sampled latency and synthetic output tokens. Tested on two different GPUs, four model variants, two model families, two attention backends, and both Poisson and bursty ShareGPT workloads, LLM-Emu closely tracks real vLLM serving behavior: TPOT and ITL stay within $4.8\%$ absolute error, E2E latency within $5.3\%$, and output throughput within $1.9\%$; TTFT is less stable, with maximum error $10.4\%$, reflecting its sensitivity to admission and queue state. These results suggest that lightweight, serving-native emulation can support practical online experimentation for LLM-serving systems. LLM-Emu is open sourced at https://github.com/AKafakA/llm-emu.