Simulation leagues: Analysis of competition formats
This work addresses competition format selection for simulation leagues like RoboCup, offering incremental improvements in ranking accuracy.
The paper tackled the problem of selecting competition formats by analyzing a 28,000-game round-robin in the RoboCupSoccer 2D simulation league, finding that a proposed new format reduced variation in team performance rankings by 75% and 67% compared to actual RoboCup 2012 and 2013 results.
The selection of an appropriate competition format is critical for both the success and credibility of any competition, both real and simulated. In this paper, the automated parallelism offered by the RoboCupSoccer 2D simulation league is leveraged to conduct a 28,000 game round-robin between the top 8 teams from RoboCup 2012 and 2013. A proposed new competition format is found to reduce variation from the resultant statistically significant team performance rankings by 75% and 67%, when compared to the actual competition results from RoboCup 2012 and 2013 respectively. These results are statistically validated by generating 10,000 random tournaments for each of the three considered formats and comparing the respective distributions of ranking discrepancy.