Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
This work addresses the need for better performance evaluation in LLM inference systems, particularly for real-time applications like chat and translation, though it is incremental as it builds on existing metrics.
The paper tackles the problem that conventional latency and throughput metrics inadequately capture user-facing performance in LLM inference systems, and proposes Etalon, a comprehensive evaluation framework including a novel fluidity-index metric, which is used to assess various open-source platforms and model-as-a-service offerings.
Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon.