Fairness Index Measures to Evaluate Bias in Biometric Recognition
Ketan Kotwal Sebastien Marcel
The demographic disparity of biometric systems has led to serious concerns
regarding their societal impact as well as applicability of such systems in
private and public domains. A quantitative evaluation of demographic fairness
is an important step towards understanding, assessment, and mitigation of
demographic bias in biometric applications. While few, existing fairness
measures are based on post-decision data (such as verification accuracy) of
biometric systems, we discuss how pre-decision data (score distributions)
provide useful insights towards demographic fairness. In this paper, we
introduce multiple measures, based on the statistical characteristics of score
distributions, for the evaluation of demographic fairness of a generic
biometric verification system. We also propose different variants for each
fairness measure depending on how the contribution from constituent demographic
groups needs to be combined towards the final measure. In each case, the
behavior of the measure has been illustrated numerically and graphically on
synthetic data. The demographic imbalance in benchmarking datasets is often
overlooked during fairness assessment. We provide a novel weighing strategy to
reduce the effect of such imbalance through a non-linear function of sample
sizes of demographic groups. The proposed measures are independent of the
biometric modality, and thus, applicable across commonly used biometric
modalities (e.g., face, fingerprint, etc.).