LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts
This provides a novel evaluation tool for AI researchers to better understand LLM intelligence in interactive scenarios, though it is incremental as it builds on existing benchmarks.
The authors tackled the problem of evaluating large language models (LLMs) beyond single metrics by introducing LLMsPark, a game theory-based benchmark that assesses strategic decision-making and social behaviors in multi-agent settings, resulting in rankings and performance differences across 15 leading LLMs.
As large language models (LLMs) advance across diverse tasks, the need for comprehensive evaluation beyond single metrics becomes increasingly important. To fully assess LLM intelligence, it is crucial to examine their interactive dynamics and strategic behaviors. We present LLMsPark, a game theory-based evaluation platform that measures LLMs' decision-making strategies and social behaviors in classic game-theoretic settings, providing a multi-agent environment to explore strategic depth. Our system cross-evaluates 15 leading LLMs (both commercial and open-source) using leaderboard rankings and scoring mechanisms. Higher scores reflect stronger reasoning and strategic capabilities, revealing distinct behavioral patterns and performance differences across models. This work introduces a novel perspective for evaluating LLMs' strategic intelligence, enriching existing benchmarks and broadening their assessment in interactive, game-theoretic scenarios. The benchmark and rankings are publicly available at https://llmsparks.github.io/.