Can ChatGPT predict article retraction based on Twitter mentions?
Er-Te Zheng Hui-Zhen Fu Zhichao Fang
Detecting problematic research articles timely is a vital task. This study
explores whether Twitter mentions of retracted articles can signal potential
problems with the articles prior to retraction, thereby playing a role in
predicting future retraction of problematic articles. A dataset comprising
3,505 retracted articles and their associated Twitter mentions is analyzed,
alongside 3,505 non-retracted articles with similar characteristics obtained
using the Coarsened Exact Matching method. The effectiveness of Twitter
mentions in predicting article retraction is evaluated by four prediction
methods, including manual labelling, keyword identification, machine learning
models, and ChatGPT. Manual labelling results indicate that there are indeed
retracted articles with their Twitter mentions containing recognizable evidence
signaling problems before retraction, although they represent only a limited
share of all retracted articles with Twitter mention data (approximately 16%).
Using the manual labelling results as the baseline, ChatGPT demonstrates
superior performance compared to other methods, implying its potential in
assisting human judgment for predicting article retraction. This study uncovers
both the potential and limitation of social media events as an early warning
system for article retraction, shedding light on a potential application of
generative artificial intelligence in promoting research integrity.