FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
Xiaotian Han Jianfeng Chi Yu Chen Qifan Wang Han Zhao Na Zou Xia Hu
This paper introduces the Fair Fairness Benchmark (\textsf{FFB}), a
benchmarking framework for in-processing group fairness methods. Ensuring
fairness in machine learning is critical for ethical and legal compliance.
However, there exist challenges in comparing and developing of fairness methods
due to inconsistencies in experimental settings, lack of accessible algorithmic
implementations, and limited extensibility of current fairness packages and
tools. To address these issues, we introduce an open-source, standardized
benchmark for evaluating in-processing group fairness methods and provide a
comprehensive analysis of state-of-the-art methods to ensure different notions
of group fairness. This work offers the following key contributions: the
provision of flexible, extensible, minimalistic, and research-oriented
open-source code; the establishment of unified fairness method benchmarking
pipelines; and extensive benchmarking, which yields key insights from
$\mathbf{45,079}$ experiments. We believe our work will significantly
facilitate the growth and development of the fairness research community. The
benchmark, including code and running logs, is available at
https://github.com/ahxt/fair_fairness_benchmark