SwarmX: Agentic Scheduling for Low-Latency Agentic Systems
For operators of large-scale GPU-CPU clusters serving agentic AI applications, SwarmX addresses the scheduling bottleneck caused by semantic-dependent inference times, achieving significant latency and throughput improvements.
SwarmX introduces agentic scheduling for low-latency agentic AI systems, using neural predictors to capture prompt and runtime features for tail-aware decisions. In production deployment and controlled experiments, it reduces tail latency by up to 61.5% and sustains up to 2x throughput compared to state-of-the-art schedulers.
Agentic AI applications compose multiple model calls and tool executions, creating new scheduling challenges for GPU-CPU clusters. Their inference time and model-call structure often depend on prompt semantics, making conventional scheduling approaches ineffective for low-latency serving. This paper presents SwarmX, a system that implements agentic scheduling for low-latency agentic applications. SwarmX uses scheduling-specific neural predictors to capture prompt, device, runtime, and target-model features; exposes distributional predictions to routers and scalers for tail-aware decisions; and provides mechanisms for predictor training and online adaptation. These predictors and mechanisms are integrated into a scheduler-agent framework that provides a common substrate for integration with existing scheduling and model-serving infrastructure. We evaluate SwarmX using production deployment (nearly one thousand GPUs and one million CPU cores) and controlled experiments on a 128-GPU testbed. Across multi-agent code generation, deep research, and multimodal agentic workflows, SwarmX reduces tail latency by up to 61.5% compared to state-of-the-art schedulers and sustains up to 2x the throughput of production schedulers under the same SLO.