IndoCulture: Exploring Geographically-Influenced Cultural Commonsense
  Reasoning Across Eleven Indonesian Provinces
Fajri Koto Rahmad Mahendra Nurul Aisyah Timothy Baldwin
Although commonsense reasoning is greatly shaped by cultural and geographical
factors, previous studies on language models have predominantly centered on
English cultures, potentially resulting in an Anglocentric bias. In this paper,
we introduce IndoCulture, aimed at understanding the influence of geographical
factors on language model reasoning ability, with a specific emphasis on the
diverse cultures found within eleven Indonesian provinces. In contrast to prior
works that relied on templates (Yin et al., 2022) and online scrapping (Fung et
al., 2024), we created IndoCulture by asking local people to manually develop
the context and plausible options based on predefined topics. Evaluations of 23
language models reveal several insights: (1) even the best open-source model
struggles with an accuracy of 53.2%, (2) models often provide more accurate
predictions for specific provinces, such as Bali and West Java, and (3) the
inclusion of location contexts enhances performance, especially in larger
models like GPT-4, emphasizing the significance of geographical context in
commonsense reasoning.