Dataset Fairness: Achievable Fairness on Your Data With Utility
  Guarantees
Muhammad Faaiz Taufiq Jean-Francois Ton Yang Liu
In machine learning fairness, training models which minimize disparity across
different sensitive groups often leads to diminished accuracy, a phenomenon
known as the fairness-accuracy trade-off. The severity of this trade-off
fundamentally depends on dataset characteristics such as dataset imbalances or
biases. Therefore using a uniform fairness requirement across datasets remains
questionable and can often lead to models with substantially low utility. To
address this, we present a computationally efficient approach to approximate
the fairness-accuracy trade-off curve tailored to individual datasets, backed
by rigorous statistical guarantees. By utilizing the You-Only-Train-Once (YOTO)
framework, our approach mitigates the computational burden of having to train
multiple models when approximating the trade-off curve. Moreover, we quantify
the uncertainty in our approximation by introducing confidence intervals around
this curve, offering a statistically grounded perspective on the acceptable
range of fairness violations for any given accuracy threshold. Our empirical
evaluation spanning tabular, image and language datasets underscores that our
approach provides practitioners with a principled framework for
dataset-specific fairness decisions across various data modalities.