Exploring Large Language Models for Classical Philology
Frederick Riemenschneider Anette Frank
Recent advances in NLP have led to the creation of powerful language models
for many languages including Ancient Greek and Latin. While prior work on
Classical languages unanimously uses BERT, in this work we create four language
models for Ancient Greek that vary along two dimensions to study their
versatility for tasks of interest for Classical languages: we explore (i)
encoder-only and encoder-decoder architectures using RoBERTa and T5 as strong
model types, and create for each of them (ii) a monolingual Ancient Greek and a
multilingual instance that includes Latin and English. We evaluate all models
on morphological and syntactic tasks, including lemmatization, which
demonstrates the added value of T5's decoding abilities. We further define two
probing tasks to investigate the knowledge acquired by models pre-trained on
Classical texts. Our experiments provide the first benchmarking analysis of
existing models of Ancient Greek. Results show that our models provide
significant improvements over the SoTA. The systematic analysis of model types
can inform future research in designing language models for Classical
languages, including the development of novel generative tasks. We make all our
models available as community resources, along with a large curated
pre-training corpus for Ancient Greek, to support the creation of a larger,
comparable model zoo for Classical Philology. Our models and resources are
available at https://github.com/Heidelberg-NLP/ancient-language-models.