GlossLM: Multilingual Pretraining for Low-Resource Interlinear Glossing
Michael Ginn Lindia Tjuatja Taiqi He Enora Rice Graham Neubig Alexis Palmer Lori Levin
A key aspect of language documentation is the creation of annotated text in a
format such as interlinear glossed text (IGT), which captures fine-grained
morphosyntactic analyses in a morpheme-by-morpheme format. Prior work has
explored methods to automatically generate IGT in order to reduce the time cost
of language analysis. However, many languages (particularly those requiring
preservation) lack sufficient IGT data to train effective models, and
crosslingual transfer has been proposed as a method to overcome this
limitation.
  We compile the largest existing corpus of IGT data from a variety of sources,
covering over 450k examples across 1.8k languages, to enable research on
crosslingual transfer and IGT generation. Then, we pretrain a large
multilingual model on a portion of this corpus, and further finetune it to
specific languages. Our model is competitive with state-of-the-art methods for
segmented data and large monolingual datasets. Meanwhile, our model outperforms
SOTA models on unsegmented text and small corpora by up to 6.6% morpheme
accuracy, demonstrating the effectiveness of crosslingual transfer for
low-resource languages.