ChatGPT as a commenter to the news: can LLMs generate human-like
  opinions?
Rayden Tseng Suzan Verberne Peter van der Putten
ChatGPT, GPT-3.5, and other large language models (LLMs) have drawn
significant attention since their release, and the abilities of these models
have been investigated for a wide variety of tasks. In this research we
investigate to what extent GPT-3.5 can generate human-like comments on Dutch
news articles. We define human likeness as `not distinguishable from human
comments', approximated by the difficulty of automatic classification between
human and GPT comments. We analyze human likeness across multiple prompting
techniques. In particular, we utilize zero-shot, few-shot and context prompts,
for two generated personas. We found that our fine-tuned BERT models can easily
distinguish human-written comments from GPT-3.5 generated comments, with none
of the used prompting methods performing noticeably better. We further analyzed
that human comments consistently showed higher lexical diversity than
GPT-generated comments. This indicates that although generative LLMs can
generate fluent text, their capability to create human-like opinionated
comments is still limited.