Fair Active Learning in Low-Data Regimes
Romain Camilleri Andrew Wagenmaker Jamie Morgenstern Lalit Jain Kevin Jamieson
In critical machine learning applications, ensuring fairness is essential to
avoid perpetuating social inequities. In this work, we address the challenges
of reducing bias and improving accuracy in data-scarce environments, where the
cost of collecting labeled data prohibits the use of large, labeled datasets.
In such settings, active learning promises to maximize marginal accuracy gains
of small amounts of labeled data. However, existing applications of active
learning for fairness fail to deliver on this, typically requiring large
labeled datasets, or failing to ensure the desired fairness tolerance is met on
the population distribution.
  To address such limitations, we introduce an innovative active learning
framework that combines an exploration procedure inspired by posterior sampling
with a fair classification subroutine. We demonstrate that this framework
performs effectively in very data-scarce regimes, maximizing accuracy while
satisfying fairness constraints with high probability. We evaluate our proposed
approach using well-established real-world benchmark datasets and compare it
against state-of-the-art methods, demonstrating its effectiveness in producing
fair models, and improvement over existing methods.