A Group Fairness Lens for Large Language Models
Guanqun Bi Lei Shen Yuqiang Xie Yanan Cao Tiangang Zhu Xiaodong He
The rapid advancement of large language models has revolutionized various
applications but also raised crucial concerns about their potential to
perpetuate biases and unfairness when deployed in social media contexts.
Evaluating LLMs' potential biases and fairness has become crucial, as existing
methods rely on limited prompts focusing on just a few groups, lacking a
comprehensive categorical perspective. In this paper, we propose evaluating LLM
biases from a group fairness lens using a novel hierarchical schema
characterizing diverse social groups. Specifically, we construct a dataset,
GFair, encapsulating target-attribute combinations across multiple dimensions.
In addition, we introduce statement organization, a new open-ended text
generation task, to uncover complex biases in LLMs. Extensive evaluations of
popular LLMs reveal inherent safety concerns. To mitigate the biases of LLM
from a group fairness perspective, we pioneer a novel chain-of-thought method
GF-Think to mitigate biases of LLMs from a group fairness perspective.
Experimental results demonstrate its efficacy in mitigating bias in LLMs to
achieve fairness.