LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
This addresses the problem of evaluating LLM agents in complex, interactive settings for AI researchers, though it is incremental as it builds on existing benchmarking efforts by adding multi-agent dynamics.
The paper tackles the lack of benchmarks for evaluating large language models (LLMs) in dynamic multi-agent environments by introducing LLMArena, a framework with seven gaming environments that assesses capabilities like spatial reasoning and team collaboration, finding that LLMs still have significant gaps, especially in opponent modeling and team collaboration.
Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code and data will be available.