Inefficiencies of Meta Agents for Agent Design
This work identifies practical limitations in automated agent design, which is incremental as it critiques existing methods without proposing a new paradigm.
The paper examined inefficiencies in meta-agents for automated agent design, finding that evolutionary approaches outperform context expansion, designed agents lack behavioral diversity, and automated design is only cost-effective for two datasets when deployed on over 15,000 examples.
Recent works began to automate the design of agentic systems using meta-agents that propose and iteratively refine new agent architectures. In this paper, we examine three key challenges in a common class of meta-agents. First, we investigate how a meta-agent learns across iterations and find that simply expanding the context with all previous agents, as proposed by previous works, performs worse than ignoring prior designs entirely. We show that the performance improves with an evolutionary approach. Second, although the meta-agent designs multiple agents during training, it typically commits to a single agent at test time. We find that the designed agents have low behavioral diversity, limiting the potential for their complementary use. Third, we assess when automated design is economically viable. We find that only in a few cases--specifically, two datasets--the overall cost of designing and deploying the agents is lower than that of human-designed agents when deployed on over 15,000 examples. In contrast, the performance gains for other datasets do not justify the design cost, regardless of scale.