Experimental Studies in General Game Playing: An Experience Report
This work identifies methodological challenges in a niche AI research area, aiming to improve reproducibility and fairness for researchers in General Game Playing.
The authors surveyed 15 years of General Game Playing research to highlight issues in reproducibility and fairness when comparing different game-playing agents and systems. They propose recommendations to address these problems and improve future experimental comparisons.
We describe nearly fifteen years of General Game Playing experimental research history in the context of reproducibility and fairness of comparisons between various GGP agents and systems designed to play games described by different formalisms. We think our survey may provide an interesting perspective of how chaotic methods were allowed when nothing better was possible. Finally, from our experience-based view, we would like to propose a few recommendations of how such specific heterogeneous branch of research should be handled appropriately in the future. The goal of this note is to point out common difficulties and problems in the experimental research in the area. We hope that our recommendations will help in avoiding them in future works and allow more fair and reproducible comparisons.