Gender inference: can chatGPT outperform common commercial tools?
Michelle Alexopoulos Kelly Lyons Kaushar Mahetaji Marcus Emmanuel Barnes Rogan Gutwillinger
An increasing number of studies use gender information to understand
phenomena such as gender bias, inequity in access and participation, or the
impact of the Covid pandemic response. Unfortunately, most datasets do not
include self-reported gender information, making it necessary for researchers
to infer gender from other information, such as names or names and country
information. An important limitation of these tools is that they fail to
appropriately capture the fact that gender exists on a non-binary scale,
however, it remains important to evaluate and compare how well these tools
perform in a variety of contexts. In this paper, we compare the performance of
a generative Artificial Intelligence (AI) tool ChatGPT with three commercially
available list-based and machine learning-based gender inference tools (Namsor,
Gender-API, and genderize.io) on a unique dataset. Specifically, we use a large
Olympic athlete dataset and report how variations in the input (e.g., first
name and first and last name, with and without country information) impact the
accuracy of their predictions. We report results for the full set, as well as
for the subsets: medal versus non-medal winners, athletes from the largest
English-speaking countries, and athletes from East Asia. On these sets, we find
that Namsor is the best traditional commercially available tool. However,
ChatGPT performs at least as well as Namsor and often outperforms it,
especially for the female sample when country and/or last name information is
available. All tools perform better on medalists versus non-medalists and on
names from English-speaking countries. Although not designed for this purpose,
ChatGPT may be a cost-effective tool for gender prediction. In the future, it
might even be possible for ChatGPT or other large scale language models to
better identify self-reported gender rather than report gender on a binary
scale.