DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the
  Knowledge of Pretrained Language Models
Amr Keleg Walid Magdy
A few benchmarking datasets have been released to evaluate the factual
knowledge of pretrained language models. These benchmarks (e.g., LAMA, and
ParaRel) are mainly developed in English and later are translated to form new
multilingual versions (e.g., mLAMA, and mParaRel). Results on these
multilingual benchmarks suggest that using English prompts to recall the facts
from multilingual models usually yields significantly better and more
consistent performance than using non-English prompts. Our analysis shows that
mLAMA is biased toward facts from Western countries, which might affect the
fairness of probing models. We propose a new framework for curating factual
triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is
built of factual triples from three pairs of contrasting cultures having a
total of 78,259 triples from 20 relation predicates. The three pairs comprise
facts representing the (Arab and Western), (Asian and Western), and (South
American and Western) countries respectively. Having a more balanced benchmark
(DLAMA-v1) supports that mBERT performs better on Western facts than
non-Western ones, while monolingual Arabic, English, and Korean models tend to
perform better on their culturally proximate facts. Moreover, both monolingual
and multilingual models tend to make a prediction that is culturally or
geographically relevant to the correct label, even if the prediction is wrong.