Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil
  Demographic Biases in Languages at Scale
Marta R. Costa-jussà Pierre Andrews Eric Smith Prangthip Hansanti Christophe Ropers Elahe Kalbassi Cynthia Gao Daniel Licht Carleigh Wood
We introduce a multilingual extension of the HOLISTICBIAS dataset, the
largest English template-based taxonomy of textual people references:
MULTILINGUALHOLISTICBIAS. This extension consists of 20,459 sentences in 50
languages distributed across all 13 demographic axes. Source sentences are
built from combinations of 118 demographic descriptors and three patterns,
excluding nonsensical combinations. Multilingual translations include
alternatives for gendered languages that cover gendered translations when there
is ambiguity in English. Our benchmark is intended to uncover demographic
imbalances and be the tool to quantify mitigations towards them.
  Our initial findings show that translation quality for EN-to-XX translations
is an average of 8 spBLEU better when evaluating with the masculine human
reference compared to feminine. In the opposite direction, XX-to-EN, we compare
the robustness of the model when the source input only differs in gender
(masculine or feminine) and masculine translations are an average of almost 4
spBLEU better than feminine. When embedding sentences to a joint multilingual
sentence representations space, we find that for most languages masculine
translations are significantly closer to the English neutral sentences when
embedded.