AI Testing Should Account for Sophisticated Strategic Behaviour
This addresses the problem of ensuring AI safety in real-world scenarios for developers and policymakers, but it is incremental as it builds on existing game-theoretic concepts.
The paper argues that AI testing must consider systems' strategic reasoning to predict deployment behavior accurately, and proposes using game theory to design evaluations, supported by examples and formal analysis.
This position paper argues for two claims regarding AI testing and evaluation. First, to remain informative about deployment behaviour, evaluations need account for the possibility that AI systems understand their circumstances and reason strategically. Second, game-theoretic analysis can inform evaluation design by formalising and scrutinising the reasoning in evaluation-based safety cases. Drawing on examples from existing AI systems, a review of relevant research, and formal strategic analysis of a stylised evaluation scenario, we present evidence for these claims and motivate several research directions.