Measuring Bias in a Ranked List using Term-based Representations
Amin Abolghasemi Leif Azzopardi Arian Askari Maarten de Rijke Suzan Verberne
In most recent studies, gender bias in document ranking is evaluated with the
NFaiRR metric, which measures bias in a ranked list based on an aggregation
over the unbiasedness scores of each ranked document. This perspective in
measuring the bias of a ranked list has a key limitation: individual documents
of a ranked list might be biased while the ranked list as a whole balances the
groups' representations. To address this issue, we propose a novel metric
called TExFAIR (term exposure-based fairness), which is based on two new
extensions to a generic fairness evaluation framework, attention-weighted
ranking fairness (AWRF). TExFAIR assesses fairness based on the term-based
representation of groups in a ranked list: (i) an explicit definition of
associating documents to groups based on probabilistic term-level associations,
and (ii) a rank-biased discounting factor (RBDF) for counting
non-representative documents towards the measurement of the fairness of a
ranked list. We assess TExFAIR on the task of measuring gender bias in passage
ranking, and study the relationship between TExFAIR and NFaiRR. Our experiments
show that there is no strong correlation between TExFAIR and NFaiRR, which
indicates that TExFAIR measures a different dimension of fairness than NFaiRR.
With TExFAIR, we extend the AWRF framework to allow for the evaluation of
fairness in settings with term-based representations of groups in documents in
a ranked list.