Evaluating Counterfactual Strategic Reasoning in Large Language Models
This work addresses the problem of understanding LLM reasoning capabilities for researchers in AI and game theory, but it is incremental as it builds on existing evaluation methods.
The study evaluated Large Language Models (LLMs) in repeated game-theoretic settings to assess if strategic performance stems from genuine reasoning or memorized patterns, finding limitations in incentive sensitivity and generalization in counterfactual variants of games like Prisoner's Dilemma and Rock-Paper-Scissors.
We evaluate Large Language Models (LLMs) in repeated game-theoretic settings to assess whether strategic performance reflects genuine reasoning or reliance on memorized patterns. We consider two canonical games, Prisoner's Dilemma (PD) and Rock-Paper-Scissors (RPS), upon which we introduce counterfactual variants that alter payoff structures and action labels, breaking familiar symmetries and dominance relations. Our multi-metric evaluation framework compares default and counterfactual instantiations, showcasing LLM limitations in incentive sensitivity, structural generalization and strategic reasoning within counterfactual environments.