Vortex: Hosting ML Inference and Knowledge Retrieval Services With Tight Latency and Throughput Requirements
This addresses the need for reliable low-latency services in AI applications, such as interactive queries and AI agents, representing an incremental improvement over existing batching-based platforms.
The paper tackles the problem of unpredictable tail latencies in ML serving platforms by introducing Vortex, an SLO-first approach for hosting ML inference and knowledge retrieval services, achieving significantly lower and more stable latencies than TorchServe and Ray Serve, often enabling over twice the request rate for a given SLO target.
There is growing interest in deploying ML inference and knowledge retrieval as services that could support both interactive queries by end users and more demanding request flows that arise from AIs integrated into a end-user applications and deployed as agents. Our central premise is that these latter cases will bring service level latency objectives (SLOs). Existing ML serving platforms use batching to optimize for high throughput, exposing them to unpredictable tail latencies. Vortex enables an SLO-first approach. For identical tasks, Vortex's pipelines achieve significantly lower and more stable latencies than TorchServe and Ray Serve over a wide range of workloads, often enabling a given SLO target at more than twice the request rate. When RDMA is available, the Vortex advantage is even more significant.