Rethinking Evaluation Metric for Probability Estimation Models Using
  Esports Data
Euihyeon Choi Jooyoung Kim Wonkyung Lee
Probability estimation models play an important role in various fields, such
as weather forecasting, recommendation systems, and sports analysis. Among
several models estimating probabilities, it is difficult to evaluate which
model gives reliable probabilities since the ground-truth probabilities are not
available. The win probability estimation model for esports, which calculates
the win probability under a certain game state, is also one of the fields being
actively studied in probability estimation. However, most of the previous works
evaluated their models using accuracy, a metric that only can measure the
performance of discrimination. In this work, we firstly investigate the Brier
score and the Expected Calibration Error (ECE) as a replacement of accuracy
used as a performance evaluation metric for win probability estimation models
in esports field. Based on the analysis, we propose a novel metric called
Balance score which is a simple yet effective metric in terms of six good
properties that probability estimation metric should have. Under the general
condition, we also found that the Balance score can be an effective
approximation of the true expected calibration error which has been imperfectly
approximated by ECE using the binning technique. Extensive evaluations using
simulation studies and real game snapshot data demonstrate the promising
potential to adopt the proposed metric not only for the win probability
estimation model for esports but also for evaluating general probability
estimation models.