Rethinking Evaluation Metric for Probability Estimation Models Using Esports Data
This work addresses the challenge of reliably assessing probability estimation models in fields like esports and beyond, offering a more accurate evaluation tool, though it is incremental as it builds on existing calibration metrics.
The paper tackles the problem of evaluating probability estimation models, particularly for esports win probability, by proposing a new metric called Balance score that approximates true expected calibration error more effectively than existing metrics like accuracy, Brier score, and ECE, demonstrating its potential through simulations and real data.
Probability estimation models play an important role in various fields, such as weather forecasting, recommendation systems, and sports analysis. Among several models estimating probabilities, it is difficult to evaluate which model gives reliable probabilities since the ground-truth probabilities are not available. The win probability estimation model for esports, which calculates the win probability under a certain game state, is also one of the fields being actively studied in probability estimation. However, most of the previous works evaluated their models using accuracy, a metric that only can measure the performance of discrimination. In this work, we firstly investigate the Brier score and the Expected Calibration Error (ECE) as a replacement of accuracy used as a performance evaluation metric for win probability estimation models in esports field. Based on the analysis, we propose a novel metric called Balance score which is a simple yet effective metric in terms of six good properties that probability estimation metric should have. Under the general condition, we also found that the Balance score can be an effective approximation of the true expected calibration error which has been imperfectly approximated by ECE using the binning technique. Extensive evaluations using simulation studies and real game snapshot data demonstrate the promising potential to adopt the proposed metric not only for the win probability estimation model for esports but also for evaluating general probability estimation models.