Pre-Trained Language Models Represent Some Geographic Populations Better
  Than Others
Jonathan Dunn Benjamin Adams Harish Tayyar Madabushi
This paper measures the skew in how well two families of LLMs represent
diverse geographic populations. A spatial probing task is used with
geo-referenced corpora to measure the degree to which pre-trained language
models from the OPT and BLOOM series represent diverse populations around the
world. Results show that these models perform much better for some populations
than others. In particular, populations across the US and the UK are
represented quite well while those in South and Southeast Asia are poorly
represented. Analysis shows that both families of models largely share the same
skew across populations. At the same time, this skew cannot be fully explained
by sociolinguistic factors, economic factors, or geographic factors. The basic
conclusion from this analysis is that pre-trained models do not equally
represent the world's population: there is a strong skew towards specific
geographic populations. This finding challenges the idea that a single model
can be used for all populations.