Can Language Models Recognize Convincing Arguments?
Paula Rescala Manoel Horta Ribeiro Tiancheng Hu Robert West
The remarkable and ever-increasing capabilities of Large Language Models
(LLMs) have raised concerns about their potential misuse for creating
personalized, convincing misinformation and propaganda. To gain insights into
LLMs' persuasive capabilities without directly engaging in experimentation with
humans, we propose studying their performance on the related task of detecting
convincing arguments. We extend a dataset by Durmus & Cardie (2018) with
debates, votes, and user traits and propose tasks measuring LLMs' ability to
(1) distinguish between strong and weak arguments, (2) predict stances based on
beliefs and demographic characteristics, and (3) determine the appeal of an
argument to an individual based on their traits. We show that LLMs perform on
par with humans in these tasks and that combining predictions from different
LLMs yields significant performance gains, even surpassing human performance.
The data and code released with this paper contribute to the crucial ongoing
effort of continuously evaluating and monitoring the rapidly evolving
capabilities and potential impact of LLMs.