GlobalBench: A Benchmark for Global Progress in Natural Language
  Processing
Yueqi Song Catherine Cui Simran Khanuja Pengfei Liu Fahim Faisal Alissa Ostapenko Genta Indra Winata Alham Fikri Aji Samuel Cahyawijaya Yulia Tsvetkov Antonios Anastasopoulos Graham Neubig
Despite the major advances in NLP, significant disparities in NLP system
performance across languages still exist. Arguably, these are due to uneven
resource allocation and sub-optimal incentives to work on less resourced
languages. To track and further incentivize the global development of equitable
language technology, we introduce GlobalBench. Prior multilingual benchmarks
are static and have focused on a limited number of tasks and languages. In
contrast, GlobalBench is an ever-expanding collection that aims to dynamically
track progress on all NLP datasets in all languages. Rather than solely
measuring accuracy, GlobalBench also tracks the estimated per-speaker utility
and equity of technology across all languages, providing a multi-faceted view
of how language technology is serving people of the world. Furthermore,
GlobalBench is designed to identify the most under-served languages, and
rewards research efforts directed towards those languages. At present, the most
under-served languages are the ones with a relatively high population, but
nonetheless overlooked by composite multilingual benchmarks (like Punjabi,
Portuguese, and Wu Chinese). Currently, GlobalBench covers 966 datasets in 190
languages, and has 1,128 system submissions spanning 62 languages.