Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraft
For multi-agent AI research, this benchmark provides a challenging testbed for real-world collaboration, but the results are incremental, showing expected limitations of current LLMs.
The paper introduces TickingCollabBench, a Minecraft benchmark for time-sensitive complementary collaboration, and finds that LLM-based agents frequently fail under dynamic environments, falling significantly short of a global-knowledge oracle.
We present TickingCollabBench, a Minecraft-based multi-agent benchmark for a novel class of time-sensitive complementary collaboration tasks. Our benchmark reflects four core characteristics of real-world collaboration: agent heterogeneity, mandatory collaboration, dynamic environments, and strict real-time constraints with failure risks. To enable this, we develop the TickingCollab framework, which supports the generation of diverse dynamic environments and abstracts Minecraft's primitive APIs to enable declarative YAML task specifications for composing these events. Building on this, we design a feasibility-aware automated benchmark generation pipeline, where an LLM drafts structurally diverse task configurations and feasibility verifier filters out invalid ones using approximate constraints. Evaluations demonstrate that lang latency and inherent difficulty of coordinating under partial observability and agent heterogeneity cause LLMs to frequently fail under dynamic environments and fall significantly short of a global-knowledge oracle.