Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Game
This addresses the problem of data saturation and leakage in static benchmarks for LLM evaluation, offering a scalable and culturally nuanced alternative, though it is incremental in its approach.
The paper tackles the need for robust evaluation of Large Language Models (LLMs) by proposing a dynamic benchmarking framework using the social deduction game Spyfall to assess multilingual and multicultural capabilities, finding that models show a significant performance gap in non-English contexts, with rankings aligning closely with Chatbot Arena.
The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings align closely with the Chatbot Arena. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The game history can be accessed here https://huggingface.co/datasets/haryoaw/cultural-spyfall.