LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale
This provides a simulation infrastructure for researchers and developers working on LLM serving systems, enabling efficient exploration of hardware-software co-designs, though it is incremental as it builds on existing simulator concepts.
The paper tackles the lack of accurate and fast simulation tools for large language model (LLM) inference serving systems by developing LLMServingSim, which models dynamic workload variations and leverages computational redundancies, achieving less than 14.7% error rate and 91.5x faster simulation speed compared to existing simulators.
Recently, there has been an extensive research effort in building efficient large language model (LLM) inference serving systems. These efforts not only include innovations in the algorithm and software domains but also constitute developments of various hardware acceleration techniques. Nevertheless, there is a lack of simulation infrastructure capable of accurately modeling versatile hardware-software behaviors in LLM serving systems without extensively extending the simulation time. This paper aims to develop an effective simulation tool, called LLMServingSim, to support future research in LLM serving systems. In designing LLMServingSim, we focus on two limitations of existing simulators: (1) they lack consideration of the dynamic workload variations of LLM inference serving due to its autoregressive nature, and (2) they incur repetitive simulations without leveraging algorithmic redundancies in LLMs. To address these limitations, LLMServingSim simulates the LLM serving in the granularity of iterations, leveraging the computation redundancies across decoder blocks and reusing the simulation results from previous iterations. Additionally, LLMServingSim provides a flexible framework that allows users to plug in any accelerator compiler-and-simulation stacks for exploring various system designs with heterogeneous processors. Our experiments demonstrate that LLMServingSim produces simulation results closely following the performance behaviors of real GPU-based LLM serving system with less than 14.7% error rate, while offering 91.5x faster simulation speed compared to existing accelerator simulators.