Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows
For practitioners building LLM-based agent systems, this work provides rigorous evidence that multi-agent workflows often do not outperform simpler single-agent setups, challenging the prevailing assumption that more agents are better.
The paper introduces BenchAgent, a controlled evaluation framework for LLM agent workflows, and finds that adding more agents rarely improves performance: only one of six multi-agent systems matched the single-agent baseline, while others underperformed by 2.56-11.29 points with higher costs.
Does adding more agents help an LLM workflow once compared systems share the same benchmark loader, tool access, answer contract, usage accounting, and trajectory logging? We introduce BenchAgent, an evaluation framework that places single-agent, fixed multi-agent (MAS), and evolving MAS workflows under one normalized execution and logging protocol. BenchAgent evaluates these substrate-internal workflows across ten reasoning, coding, and tool-use benchmarks with GPT-4.1, and separately reports a Protocol-Aligned External (PAE) GAIA study of a runtime-generated workflow. Under SI conditions, at most one of six tested MAS exceeds the matched single-agent anchor on benchmark-balanced average accuracy: EvoAgent lies within the Wilson one-run guidance, while the remaining five trail by 2.56-11.29 points and occupy more expensive accuracy-cost trade-offs. On the PAE GAIA snapshot, a Claude-Code-style runtime workflow reaches 66.72% overall and 69.23% on Level 3, more than 20 points above the strongest non-Claude baseline, Jarvis, a fixed MAS.