Native Language Identification with Large Language Models
Wei Zhang Alexandre Salle
We present the first experiments on Native Language Identification (NLI)
using LLMs such as GPT-4. NLI is the task of predicting a writer's first
language by analyzing their writings in a second language, and is used in
second language acquisition and forensic linguistics. Our results show that GPT
models are proficient at NLI classification, with GPT-4 setting a new
performance record of 91.7% on the benchmark TOEFL11 test set in a zero-shot
setting. We also show that unlike previous fully-supervised settings, LLMs can
perform NLI without being limited to a set of known classes, which has
practical implications for real-world applications. Finally, we also show that
LLMs can provide justification for their choices, providing reasoning based on
spelling errors, syntactic patterns, and usage of directly translated
linguistic patterns.