Towards Better Fairness-Utility Trade-off: A Comprehensive
  Measurement-Based Reinforcement Learning Framework
Simiao Zhang Jitao Bai Menghong Guan Yihao Huang Yueling Zhang Jun Sun Geguang Pu
Machine learning is widely used to make decisions with societal impact such
as bank loan approving, criminal sentencing, and resume filtering. How to
ensure its fairness while maintaining utility is a challenging but crucial
issue. Fairness is a complex and context-dependent concept with over 70
different measurement metrics. Since existing regulations are often vague in
terms of which metric to use and different organizations may prefer different
fairness metrics, it is important to have means of improving fairness
comprehensively. Existing mitigation techniques often target at one specific
fairness metric and have limitations in improving multiple notions of fairness
simultaneously. In this work, we propose CFU (Comprehensive Fairness-Utility),
a reinforcement learning-based framework, to efficiently improve the
fairness-utility trade-off in machine learning classifiers. A comprehensive
measurement that can simultaneously consider multiple fairness notions as well
as utility is established, and new metrics are proposed based on an in-depth
analysis of the relationship between different fairness metrics. The reward
function of CFU is constructed with comprehensive measurement and new metrics.
We conduct extensive experiments to evaluate CFU on 6 tasks, 3 machine learning
models, and 15 fairness-utility measurements. The results demonstrate that CFU
can improve the classifier on multiple fairness metrics without sacrificing its
utility. It outperforms all state-of-the-art techniques and has witnessed a
37.5% improvement on average.