Generative AI Text Classification using Ensemble LLM Approaches
Harika Abburi Michael Suesserman Nirmala Pudota Balaji Veeramani Edward Bowen Sanmitra Bhattacharya
Large Language Models (LLMs) have shown impressive performance across a
variety of Artificial Intelligence (AI) and natural language processing tasks,
such as content creation, report generation, etc. However, unregulated malign
application of these models can create undesirable consequences such as
generation of fake news, plagiarism, etc. As a result, accurate detection of
AI-generated language can be crucial in responsible usage of LLMs. In this
work, we explore 1) whether a certain body of text is AI generated or written
by human, and 2) attribution of a specific language model in generating a body
of text. Texts in both English and Spanish are considered. The datasets used in
this study are provided as part of the Automated Text Identification
(AuTexTification) shared task. For each of the research objectives stated
above, we propose an ensemble neural model that generates probabilities from
different pre-trained LLMs which are used as features to a Traditional Machine
Learning (TML) classifier following it. For the first task of distinguishing
between AI and human generated text, our model ranked in fifth and thirteenth
place (with macro $F1$ scores of 0.733 and 0.649) for English and Spanish
texts, respectively. For the second task on model attribution, our model ranked
in first place with macro $F1$ scores of 0.625 and 0.653 for English and
Spanish texts, respectively.