ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition
This provides a scalable evaluation framework for LLM developers and researchers, though it is incremental as it builds on existing game-based approaches.
The paper tackles the problem of evaluating Large Language Models (LLMs) by introducing ZeroSumEval, a dynamic and extensible framework that uses competitive games like Capture the Flag and chess to assess capabilities such as strategic reasoning and safety, resulting in a standardized system for scalable evaluation.
We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses a diverse suite of games, including security challenges (Capture the Flag), classic board games (chess), and knowledge tests (MathQuiz). These games are designed to evaluate a range of capabilities such as strategic reasoning, planning, knowledge application, safety, and adaptability. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework for easily implementing games and leverages DSPy to provide a better abstraction for LLM player strategies.