Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference
This addresses system-level efficiency challenges for large-scale AI agent inference in datacenters, proposing hardware co-design and disaggregation solutions.
The paper identifies memory capacity, bandwidth, and interconnect bottlenecks in AI agent inference and introduces Operational Intensity (OI) and Capacity Footprint (CF) metrics to analyze these issues, showing that long context KV cache makes decode highly memory-bound across various agent workflows and model types.
AI agent inference is driving an inference heavy datacenter future and exposes bottlenecks beyond compute - especially memory capacity, memory bandwidth and high-speed interconnect. We introduce two metrics - Operational Intensity (OI) and Capacity Footprint (CF) - that jointly explain regimes the classic roofline analysis misses, including the memory capacity wall. Across agentic workflows (chat, coding, web use, computer use) and base model choices (GQA/MLA, MoE, quantization), OI/CF can shift dramatically, with long context KV cache making decode highly memory bound. These observations motivate disaggregated serving and system level heterogeneity: specialized prefill and decode accelerators, broader scale up networking, and decoupled compute-memory enabled by optical I/O. We further hypothesize agent-hardware co design, multiple inference accelerators within one system, and high bandwidth, large capacity memory disaggregation as foundations for adaptation to evolving OI/CF. Together, these directions chart a path to sustain efficiency and capability for large scale agentic AI inference.