HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis
Nafis Irtiza Tripto Adaku Uchendu Thai Le Mattia Setzu Fosca Giannotti Dongwon Lee
Authorship Analysis, also known as stylometry, has been an essential aspect
of Natural Language Processing (NLP) for a long time. Likewise, the recent
advancement of Large Language Models (LLMs) has made authorship analysis
increasingly crucial for distinguishing between human-written and AI-generated
texts. However, these authorship analysis tasks have primarily been focused on
written texts, not considering spoken texts. Thus, we introduce the largest
benchmark for spoken texts - HANSEN (Human ANd ai Spoken tExt beNchmark).
HANSEN encompasses meticulous curation of existing speech datasets accompanied
by transcripts, alongside the creation of novel AI-generated spoken text
datasets. Together, it comprises 17 human datasets, and AI-generated spoken
texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To
evaluate and demonstrate the utility of HANSEN, we perform Authorship
Attribution (AA) & Author Verification (AV) on human-spoken datasets and
conducted Human vs. AI spoken text detection using state-of-the-art (SOTA)
models. While SOTA methods, such as, character ngram or Transformer-based
model, exhibit similar AA & AV performance in human-spoken datasets compared to
written ones, there is much room for improvement in AI-generated spoken text
detection. The HANSEN benchmark is available at:
https://huggingface.co/datasets/HANSEN-REPO/HANSEN.