When Should We Orchestrate Multiple Agents?
This work addresses the challenge of optimizing multi-agent systems for researchers and practitioners, offering a foundational approach to avoid overestimating performance and underestimating costs, though it is incremental in refining existing orchestration strategies.
The paper tackles the problem of when to orchestrate multiple agents by developing a framework that accounts for realistic constraints like inference costs and availability, showing theoretically that orchestration is effective only with performance or cost differentials and empirically demonstrating its utility in simulated agent selection, Rogers' Paradox, and a question-answer user study.
Strategies for orchestrating the interactions between multiple agents, both human and artificial, can wildly overestimate performance and underestimate the cost of orchestration. We design a framework to orchestrate agents under realistic conditions, such as inference costs or availability constraints. We show theoretically that orchestration is only effective if there are performance or cost differentials between agents. We then empirically demonstrate how orchestration between multiple agents can be helpful for selecting agents in a simulated environment, picking a learning strategy in the infamous Rogers' Paradox from social science, and outsourcing tasks to other agents during a question-answer task in a user study.