Evaluating the Performance of ChatGPT for Spam Email Detection
Yuwei Wu Shijing Si Yugui Zhang Jiawen Gu Jedrek Wosik
Email continues to be a pivotal and extensively utilized communication medium
within professional and commercial domains. Nonetheless, the prevalence of spam
emails poses a significant challenge for users, disrupting their daily routines
and diminishing productivity. Consequently, accurately identifying and
filtering spam based on content has become crucial for cybersecurity. Recent
advancements in natural language processing, particularly with large language
models like ChatGPT, have shown remarkable performance in tasks such as
question answering and text generation. However, its potential in spam
identification remains underexplored. To fill in the gap, this study attempts
to evaluate ChatGPT's capabilities for spam identification in both English and
Chinese email datasets. We employ ChatGPT for spam email detection using
in-context learning, which requires a prompt instruction and a few
demonstrations. We also investigate how the training example size affects the
performance of ChatGPT. For comparison, we also implement five popular
benchmark methods, including naive Bayes, support vector machines (SVM),
logistic regression (LR), feedforward dense neural networks (DNN), and BERT
classifiers. Though extensive experiments, the performance of ChatGPT is
significantly worse than deep supervised learning methods in the large English
dataset, while it presents superior performance on the low-resourced Chinese
dataset, even outperforming BERT in this case.