Uncertainty-based Fairness Measures
Selim Kuzucu Jiaee Cheong Hatice Gunes Sinan Kalkan
Unfair predictions of machine learning (ML) models impede their broad
acceptance in real-world settings. Tackling this arduous challenge first
necessitates defining what it means for an ML model to be fair. This has been
addressed by the ML community with various measures of fairness that depend on
the prediction outcomes of the ML models, either at the group level or the
individual level. These fairness measures are limited in that they utilize
point predictions, neglecting their variances, or uncertainties, making them
susceptible to noise, missingness and shifts in data. In this paper, we first
show that an ML model may appear to be fair with existing point-based fairness
measures but biased against a demographic group in terms of prediction
uncertainties. Then, we introduce new fairness measures based on different
types of uncertainties, namely, aleatoric uncertainty and epistemic
uncertainty. We demonstrate on many datasets that (i) our uncertainty-based
measures are complementary to existing measures of fairness, and (ii) they
provide more insights about the underlying issues leading to bias.