Well-Calibrated Probabilistic Predictive Maintenance using Venn-Abers
Ulf Johansson Tuwe Löfström Cecilia Sönströd
When using machine learning for fault detection, a common problem is the fact
that most data sets are very unbalanced, with the minority class (a fault)
being the interesting one. In this paper, we investigate the usage of
Venn-Abers predictors, looking specifically at the effect on the minority class
predictions. A key property of Venn-Abers predictors is that they output
well-calibrated probability intervals. In the experiments, we apply Venn-Abers
calibration to decision trees, random forests and XGBoost models, showing how
both overconfident and underconfident models are corrected. In addition, the
benefit of using the valid probability intervals produced by Venn-Abers for
decision support is demonstrated. When using techniques producing opaque
underlying models, e.g., random forest and XGBoost, each prediction will
consist of not only the label, but also a valid probability interval, where the
width is an indication of the confidence in the estimate. Adding Venn-Abers on
top of a decision tree allows inspection and analysis of the model, to
understand both the underlying relationship, and finding out in which parts of
feature space that the model is accurate and/or confident.