Pie: A Programmable Serving System for Emerging LLM Applications
This addresses the problem of inefficient serving for emerging LLM applications with agentic workflows, offering a flexible solution that is incremental in its approach.
The paper tackles the inflexibility of existing LLM serving systems for diverse applications by introducing Pie, a programmable system that decomposes the generation loop into fine-grained handlers, resulting in 1.3x-3.4x higher latency and throughput on agentic workflows with 3-12% overhead on standard tasks.
Emerging large language model (LLM) applications involve diverse reasoning strategies and agentic workflows, straining the capabilities of existing serving systems built on a monolithic token generation loop. This paper introduces Pie, a programmable LLM serving system designed for flexibility and efficiency. Pie decomposes the traditional generation loop into fine-grained service handlers exposed via an API and delegates control of the generation process to user-provided programs, called inferlets. This enables applications to implement new KV cache strategies, bespoke generation logic, and seamlessly integrate computation and I/O-entirely within the application, without requiring modifications to the serving system. Pie executes inferlets using WebAssembly, benefiting from its lightweight sandboxing. Our evaluation shows Pie matches state-of-the-art performance on standard tasks (3-12% latency overhead) while significantly improving latency and throughput (1.3x-3.4x higher) on agentic workflows by enabling application-specific optimizations.