D3CODE: Disentangling Disagreements in Data across Cultures on
  Offensiveness Detection and Evaluation
Aida Mostafazadeh Davani Mark Díaz Dylan Baker Vinodkumar Prabhakaran
While human annotations play a crucial role in language technologies,
annotator subjectivity has long been overlooked in data collection. Recent
studies that have critically examined this issue are often situated in the
Western context, and solely document differences across age, gender, or racial
groups. As a result, NLP research on subjectivity have overlooked the fact that
individuals within demographic groups may hold diverse values, which can
influence their perceptions beyond their group norms. To effectively
incorporate these considerations into NLP pipelines, we need datasets with
extensive parallel annotations from various social and cultural groups. In this
paper we introduce the \dataset dataset: a large-scale cross-cultural dataset
of parallel annotations for offensive language in over 4.5K sentences annotated
by a pool of over 4k annotators, balanced across gender and age, from across 21
countries, representing eight geo-cultural regions. The dataset contains
annotators' moral values captured along six moral foundations: care, equality,
proportionality, authority, loyalty, and purity. Our analyses reveal
substantial regional variations in annotators' perceptions that are shaped by
individual moral values, offering crucial insights for building pluralistic,
culturally sensitive NLP models.