Orla: A Library for Serving LLM-Based Multi-Agent Systems
This work addresses the problem for developers of LLM-based agentic applications by providing a library to simplify system construction, though it is incremental as it builds on existing LLM serving engines.
The authors tackled the challenge of building and running LLM-based multi-agent systems by introducing Orla, a library that separates request execution from workflow-level policy, demonstrating improvements in latency, cost, and time-to-first-token compared to a baseline.
We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper, which assigns each stage to an appropriate model and backend; a workflow orchestrator, which schedules stages and manages their resources and context; and a memory manager, which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token.