MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?
For researchers and practitioners building multi-agent LLM systems, this paper provides the first comprehensive benchmark showing when prompt optimization helps and when it doesn't, clarifying a previously unclear trade-off.
This work systematically studies system-prompt optimization in multi-agent LLM systems, revealing significant potential gains (e.g., up to 30% improvement on some tasks) while showing that benefits are highly dependent on task, workflow, and team size, with diminishing returns in larger teams.
Multi-agent systems (MAS) offer a scalable path forward for agentic AI, comprising multiple LLM-based agents, each assigned a system prompt and a position within a workflow that governs inter-agent coordination and output aggregation. System prompts thus form a critical and accessible optimization surface: they specify agents' roles and behaviors, enabling system-level improvements without model finetuning. Although prompt optimization has shown substantial potential for single LLMs, extending it to MAS poses distinct challenges, notably an exponentially growing search space. It remains unclear whether, when, and by how much prompt optimization improves MAS performance, and how sensitive such gains are to system configuration. In this work, we systematically study system-prompt optimization across a broad range of MAS setups varying in task, workflow, communication protocol, and team size, benchmarking two prompt optimizers that naturally extend state-of-the-art single-agent methods. The results reveal its potential to unlock significant gains while exposing open challenges, characterizing when and how much prompt optimization helps across diverse MAS settings.