Performance Characterization of Expert Router for Scalable LLM Inference
This addresses the problem of efficient and scalable LLM deployment for users in scientific and industrial domains, though it appears incremental as it builds on existing routing mechanisms.
The paper tackles the challenge of scaling LLM inference by introducing Expert Router, a routing architecture that directs prompts to specialized expert models, finding that it introduces minimal latency overhead and that smaller expert models maintain competitive performance across a wider range of concurrent users compared to baseline models.
Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of LLMs' high computational and memory demands. Specialized models optimized for specific tasks can be combined through a routing mechanism to address these challenges, creating a modular inference system. This paper introduces Expert Router, a scalable routing architecture that directs prompts to specialized expert models. We characterize multiple Expert Router configurations, including different LLama 3 models with quantized and non-quantized weights under up to 1,000 concurrent users. Our findings reveal that Expert Router introduces minimal latency overhead, with the configuration of expert models being a dominating factor in performance outcomes. High-parameter expert models deliver stable throughput and latency under moderate concurrency levels. In contrast, smaller expert models maintain competitive performance across a wider range of concurrent users compared to tensor-parallelized baseline models. This highlights the potential of Expert Router for efficient and scalable LLM deployment.