Native Language Identification with Big Bird Embeddings
Sergey Kramp Giovanni Cassani Chris Emmery
Native Language Identification (NLI) intends to classify an author's native
language based on their writing in another language. Historically, the task has
heavily relied on time-consuming linguistic feature engineering, and
transformer-based NLI models have thus far failed to offer effective, practical
alternatives. The current work investigates if input size is a limiting factor,
and shows that classifiers trained using Big Bird embeddings outperform
linguistic feature engineering models by a large margin on the Reddit-L2
dataset. Additionally, we provide further insight into input length
dependencies, show consistent out-of-sample performance, and qualitatively
analyze the embedding space. Given the effectiveness and computational
efficiency of this method, we believe it offers a promising avenue for future
NLI work.