UnMASKed: Quantifying Gender Biases in Masked Language Models through
  Linguistically Informed Job Market Prompts
Iñigo Parra
Language models (LMs) have become pivotal in the realm of technological
advancements. While their capabilities are vast and transformative, they often
include societal biases encoded in the human-produced datasets used for their
training. This research delves into the inherent biases present in masked
language models (MLMs), with a specific focus on gender biases. This study
evaluated six prominent models: BERT, RoBERTa, DistilBERT, BERT-multilingual,
XLM-RoBERTa, and DistilBERT-multilingual. The methodology employed a novel
dataset, bifurcated into two subsets: one containing prompts that encouraged
models to generate subject pronouns in English, and the other requiring models
to return the probabilities of verbs, adverbs, and adjectives linked to the
prompts' gender pronouns. The analysis reveals stereotypical gender alignment
of all models, with multilingual variants showing comparatively reduced biases.