AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
This provides a flexible tool for researchers across disciplines to test LLM capacities, though it is incremental as it builds on existing simulation-based evaluation ideas.
The paper tackles the problem of evaluating large language models (LLMs) by proposing AgentSims, an open-source sandbox for task-based evaluation in simulated environments, which addresses limitations like constrained abilities and unobjective metrics in existing methods.
With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question. Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. We present AgentSims, an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in. Researchers can build their evaluation tasks by adding agents and buildings on an interactive GUI or deploy and test new support mechanisms, i.e. memory, planning and tool-use systems, by a few lines of codes. Our demo is available at https://agentsims.com .