Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
For systems researchers and engineers building LLM serving infrastructure, this paper provides empirical evidence and a framework to guide the design of agentic serving systems, though it is primarily a position/vision paper with preliminary experiments.
The paper introduces Aries, a full-stack experimentation framework for agentic serving systems, and uses it to reveal that token-centric metrics miss non-inference bottlenecks, retaining extra context yields diminishing accuracy returns while reducing serving capacity, and tool sandboxes exhibit idle/burst patterns where snapshot-based suspension is costly. The authors propose a vision for agent-native serving systems with trajectory-level metrics, adaptive context management, and elastic sandbox resources.
Autonomous agents challenge conventional LLM serving by coupling repeated inference with persistent context and sandboxed tool execution. We present Aries, a full-stack experimentation framework that separates task semantics from execution configurations, reconstructs cross-component agent trajectories with correlated system telemetry, and exposes stateful tool execution through a consistent interface across heterogeneous sandbox substrates. We use Aries to conduct reproducible experiments on open agent harnesses and benchmarks. We complement these experiments with production traces from a commercial platform, grounding low-level systems research in observed production behavior. Our results show that (1) token-centric metrics miss non-inference bottlenecks, (2) retaining additional context yields diminishing accuracy benefits while reducing serving capacity, and (3) tool sandboxes alternate between long idle periods and short resource bursts, while current snapshot-based state management makes aggressive suspension costly. A complementary security analysis further highlights the need to reduce the sandbox attack surface. We then discuss the vision for agent-native serving systems designed around trajectory-level metrics, adaptive context management, elastic sandbox resource management, and sandboxes with minimized attack surface.