Fairness in Serving Large Language Models
This addresses fairness and resource efficiency issues for providers and users of high-demand LLM services like ChatGPT and BARD, representing an incremental improvement over existing fair scheduling methods.
The paper tackles the problem of ensuring fairness in large language model (LLM) inference services, where existing rate limits cause under-utilization and poor client experience, by introducing a fairness definition based on token processing costs and proposing the Virtual Token Counter (VTC) scheduling algorithm, which achieves a proven 2x tight upper bound on service difference between clients.
High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading. To ensure that all client requests are processed fairly, most major LLM inference services have request rate limits, to ensure that no client can dominate the request queue. However, this rudimentary notion of fairness also results in under-utilization of the resources and poor client experience when there is spare capacity. While there is a rich literature on fair scheduling, serving LLMs presents new challenges due to their unpredictable request lengths and their unique batching characteristics on parallel accelerators. This paper introduces the definition of LLM serving fairness based on a cost function that accounts for the number of input and output tokens processed. To achieve fairness in serving, we propose a novel scheduling algorithm, the Virtual Token Counter (VTC), a fair scheduler based on the continuous batching mechanism. We prove a 2x tight upper bound on the service difference between two backlogged clients, adhering to the requirement of work-conserving. Through extensive experiments, we demonstrate the superior performance of VTC in ensuring fairness, especially in contrast to other baseline methods, which exhibit shortcomings under various conditions. The reproducible code is available at https://github.com/Ying1123/VTC-artifact