Which LLM Multi-Agent Protocol to Choose?
This work addresses the under-evaluated issue of protocol selection for developers and researchers in multi-agent systems, offering a standardized benchmark and router to improve performance and reliability, though it is incremental in building on existing protocols.
The paper tackles the problem of selecting communication protocols in large-scale multi-agent systems by introducing ProtocolBench, a benchmark that shows protocol choice can affect completion time by up to 36.5% and latency by 3.48 seconds, and ProtocolRouter, a learnable router that reduces recovery time by up to 18.1% compared to baselines.
As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.