AsymPuzl: An Asymmetric Puzzle for multi-agent cooperation
This provides a controlled testbed for studying communication and coordination in multi-agent LLM systems, addressing a gap in existing open-ended role-play setups.
The researchers tackled the problem of evaluating multi-agent cooperation in LLMs by introducing AsymPuzl, a two-agent puzzle environment with information asymmetry, and found that strong models like GPT-5 and Claude-4.0 reliably solve puzzles by sharing information in two turns, while weaker models often fail due to poor communication strategies.
Large Language Model (LLM) agents are increasingly studied in multi-turn, multi-agent scenarios, yet most existing setups emphasize open-ended role-play rather than controlled evaluation. We introduce AsymPuzl, a minimal but expressive two-agent puzzle environment designed to isolate communication under information asymmetry. Each agent observes complementary but incomplete views of a symbolic puzzle and must exchange messages to solve it cooperatively. Using a diverse set of current-generation and open-source LLMs, we show that (i) strong models such as GPT-5 and Claude-4.0 reliably converge across puzzle sizes on the solution by sharing complete information in two turns, (ii) weaker models often ignore partner messages or over-correct their hypotheses, and (iii) feedback design is non-trivial: simple self-feedback improves success rates, while detailed joint feedback can hurt performance. These findings show that even in simple cooperative tasks, LLM communication strategies diverge and depend on the granularity of feedback signals. AsymPuzl thus provides a testbed for probing the limits of multi-turn cooperation and opens avenues for studying coordination mechanisms.