Second-Order Uncertainty Quantification: A Distance-Based Approach
Yusuf Sale Viktor Bengs Michele Caprio Eyke Hüllermeier
In the past couple of years, various approaches to representing and
quantifying different types of predictive uncertainty in machine learning,
notably in the setting of classification, have been proposed on the basis of
second-order probability distributions, i.e., predictions in the form of
distributions on probability distributions. A completely conclusive solution
has not yet been found, however, as shown by recent criticisms of commonly used
uncertainty measures associated with second-order distributions, identifying
undesirable theoretical properties of these measures. In light of these
criticisms, we propose a set of formal criteria that meaningful uncertainty
measures for predictive uncertainty based on second-order distributions should
obey. Moreover, we provide a general framework for developing uncertainty
measures to account for these criteria, and offer an instantiation based on the
Wasserstein distance, for which we prove that all criteria are satisfied.