Performance Metrics for Probabilistic Ordinal Classifiers
Adrian Galdran
Ordinal classification models assign higher penalties to predictions further
away from the true class. As a result, they are appropriate for relevant
diagnostic tasks like disease progression prediction or medical image grading.
The consensus for assessing their categorical predictions dictates the use of
distance-sensitive metrics like the Quadratic-Weighted Kappa score or the
Expected Cost. However, there has been little discussion regarding how to
measure performance of probabilistic predictions for ordinal classifiers. In
conventional classification, common measures for probabilistic predictions are
Proper Scoring Rules (PSR) like the Brier score, or Calibration Errors like the
ECE, yet these are not optimal choices for ordinal classification. A PSR named
Ranked Probability Score (RPS), widely popular in the forecasting field, is
more suitable for this task, but it has received no attention in the image
analysis community. This paper advocates the use of the RPS for image grading
tasks. In addition, we demonstrate a counter-intuitive and questionable
behavior of this score, and propose a simple fix for it. Comprehensive
experiments on four large-scale biomedical image grading problems over three
different datasets show that the RPS is a more suitable performance metric for
probabilistic ordinal predictions. Code to reproduce our experiments can be
found at https://github.com/agaldran/prob_ord_metrics .