A Workflow-Aware Serving Layer for Agentic Applications
This work addresses the critical gap between model-serving engines and agent frameworks for emerging agentic AI workloads, enabling efficient and quality-aware execution of complex workflows under dynamic serving conditions.
Dyserve introduces a workflow-aware serving layer for agentic applications that jointly optimizes model, verifier, and backend selection across heterogeneous hardware, achieving up to 2x reduction in end-to-end latency while maintaining output quality compared to existing model-serving and agent framework approaches.
Agentic AI applications form an emerging serving workload in which a request creates a workflow: a directed acyclic graph of LLM and tool calls that exposes per-node model choices and optional quality operators such as verifiers. This workload falls between two existing layers. Model-serving engines execute individual calls efficiently but cannot see workflow structure, while agent frameworks fix the workflow but cannot see backend load, so neither jointly chooses each node's model, verifier, and backend under serving-time conditions. We present Dyserve, a workflow-aware serving layer that fills this gap. Dyserve compiles each workflow's per-node model and verifier choices in one integer linear program (ILP) over a heterogeneous backend pool, priced by skill-conditioned offline profiles that transfer across workflows. This couples with hardware entering only through per-model throughput sweeps, and is weighted to concentrate strong models and verification on the nodes whose errors propagate the furthest. Because no single latency-quality preference fits every workload mix, Dyserve pre-solves the program at several pressure levels at admission and shifts a workflow's uncommitted suffix among these strategies under load, keeping the solver off the load-shift path; a failed tool call triggers a one-time residual re-solve that preserves committed work.