RENOVI: A Benchmark Towards Remediating Norm Violations in
  Socio-Cultural Conversations
Haolan Zhan Zhuang Li Xiaoxi Kang Tao Feng Yuncheng Hua Lizhen Qu Yi Ying Mei Rianto Chandra Kelly Rosalin Jureynolds Jureynolds Suraj Sharma Shilin Qu Linhao Luo Lay-Ki Soon Zhaleh Semnani Azad Ingrid Zukerman Gholamreza Haffari
Norm violations occur when individuals fail to conform to culturally accepted
behaviors, which may lead to potential conflicts. Remediating norm violations
requires social awareness and cultural sensitivity of the nuances at play. To
equip interactive AI systems with a remediation ability, we offer ReNoVi - a
large-scale corpus of 9,258 multi-turn dialogues annotated with social norms,
as well as define a sequence of tasks to help understand and remediate norm
violations step by step. ReNoVi consists of two parts: 512 human-authored
dialogues (real data), and 8,746 synthetic conversations generated by ChatGPT
through prompt learning. While collecting sufficient human-authored data is
costly, synthetic conversations provide suitable amounts of data to help
mitigate the scarcity of training data, as well as the chance to assess the
alignment between LLMs and humans in the awareness of social norms. We thus
harness the power of ChatGPT to generate synthetic training data for our task.
To ensure the quality of both human-authored and synthetic data, we follow a
quality control protocol during data collection. Our experimental results
demonstrate the importance of remediating norm violations in socio-cultural
conversations, as well as the improvement in performance obtained from
synthetic data.