Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
For LLM serving systems handling mixed latency-critical and background requests, Nitsum addresses the challenge of maximizing goodput under varying workloads.
Nitsum improves SLO-compliant goodput for multi-tenant LLM serving by dynamically adapting tensor parallelism, achieving up to 5.3x improvement over state-of-the-art systems.
LLM serving is increasingly multi-tenant: the same deployment must handle latency-critical interactive requests and more relaxed background workloads under a fixed GPU budget. This creates a tiered-SLO setting where maximizing overall goodput (requests that satisfy both TTFT and TPOT targets) is challenging because workload mix, request lengths, and load intensity vary over time. Existing systems mainly optimize request-level controls (e.g., queuing and batching) while keeping execution configuration largely static, which limits adaptation under multi-tier contention. We present Nitsum, a distributed LLM serving system that treats tensor parallelism (TP) as a first-class runtime control surface rather than a static deployment choice. Nitsum jointly optimizes TP level, prefill/decode GPU split, and request scheduling. To make frequent TP adaptation practical, Nitsum introduces TP-aware weight reuse and fast KV migration. Experiments on real traces and targeted microbenchmarks show that Nitsum improves SLO-compliant goodput over SoTA by up to 5.3 times.