SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes
Mukul Bhutani Kevin Robinson Vinodkumar Prabhakaran Shachi Dave Sunipa Dev
While generative multilingual models are rapidly being deployed, their safety
and fairness evaluations are largely limited to resources collected in English.
This is especially problematic for evaluations targeting inherently
socio-cultural phenomena such as stereotyping, where it is important to build
multi-lingual resources that reflect the stereotypes prevalent in respective
language communities. However, gathering these resources, at scale, in varied
languages and regions pose a significant challenge as it requires broad
socio-cultural knowledge and can also be prohibitively expensive. To overcome
this critical gap, we employ a recently introduced approach that couples LLM
generations for scale with culturally situated validations for reliability, and
build SeeGULL Multilingual, a global-scale multilingual dataset of social
stereotypes, containing over 25K stereotypes, spanning 20 languages, with human
annotations across 23 regions, and demonstrate its utility in identifying gaps
in model evaluations. Content warning: Stereotypes shared in this paper can be
offensive.