Can Language Model Moderators Improve the Health of Online Discourse?
Hyundong Cho Shuai Liu Taiwei Shi Darpan Jain Basem Rizk Yuyang Huang Zixun Lu Nuan Wen Jonathan Gratch Emilio Ferrara Jonathan May
Human moderation of online conversation is essential to maintaining civility
and focus in a dialogue, but is challenging to scale and harmful to moderators.
The inclusion of sophisticated natural language generation modules as a force
multiplier aid moderators is a tantalizing prospect, but adequate evaluation
approaches have so far been elusive. In this paper, we establish a systematic
definition of conversational moderation effectiveness through a
multidisciplinary lens that incorporates insights from social science. We then
propose a comprehensive evaluation framework that uses this definition to asses
models' moderation capabilities independently of human intervention. With our
framework, we conduct the first known study of conversational dialogue models
as moderators, finding that appropriately prompted models can provide specific
and fair feedback on toxic behavior but struggle to influence users to increase
their levels of respect and cooperation.