SimulBench: Evaluating Language Models with Creative Simulation Tasks
This work addresses the need for better evaluation of LLMs in interactive simulation scenarios, which is incremental as it builds on existing benchmarking methods.
The authors tackled the problem of evaluating large language models (LLMs) on creative simulation tasks by introducing SimulBench, a benchmark that uses a fixed LLM as a user agent to collect dialogues and GPT-4 as an evaluator, showing that GPT-4-turbo outperforms LLaMA-3-70b-Chat by 18.55% more cases.
We introduce SimulBench, a benchmark designed to evaluate large language models (LLMs) across a diverse collection of creative simulation scenarios, such as acting as a Linux terminal or playing text games with users. While these simulation tasks serve as effective measures of an LLM's general intelligence, they are seldom incorporated into existing benchmarks. A major challenge is to develop an evaluation framework for testing different LLMs fairly while preserving the multi-round interactive nature of simulation tasks between users and AI. To tackle this issue, we suggest using a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks. Then, challenging dialogue scripts are extracted for evaluating different target LLMs. To facilitate automatic assessment on \DataName{}, GPT-4 is employed as the evaluator, tasked with reviewing the quality of the final response generated by the target LLMs given multi-turn dialogue scripts. Our comprehensive experiments indicate that these simulation tasks continue to pose a significant challenge with their unique natures and show the gap between proprietary models and the most advanced open LLMs. For example, GPT-4-turbo outperforms LLaMA-3-70b-Chat on 18.55\% more cases.