CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale
This provides a critical foundation for advancing research in scalable multi-agent Agentic AI, addressing a gap for researchers and developers in AI and multi-agent systems, though it is incremental as it builds on existing simulation platforms.
The paper tackles the lack of benchmarks for evaluating large-scale, robust multi-agent systems in complex real-world tasks by introducing CREW-Wildfire, an open-source benchmark with procedurally generated wildfire response scenarios, and it reveals significant performance gaps in existing LLM-based frameworks, highlighting challenges in coordination and planning.
Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabilities in complex, dynamic, real-world tasks. Existing environments typically focus on small-scale, fully observable, or low-complexity domains, limiting their utility for developing and assessing next-generation multi-agent Agentic AI frameworks. We introduce CREW-Wildfire, an open-source benchmark designed to close this gap. Built atop the human-AI teaming CREW simulation platform, CREW-Wildfire offers procedurally generated wildfire response scenarios featuring large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives. The environment supports both low-level control and high-level natural language interactions through modular Perception and Execution modules. We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon planning under uncertainty. By providing more realistic complexity, scalable architecture, and behavioral evaluation metrics, CREW-Wildfire establishes a critical foundation for advancing research in scalable multi-agent Agentic intelligence. All code, environments, data, and baselines will be released to support future research in this emerging domain.