Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge
  Distillation
Kevin Krahn Derrick Tate Andrew C. Lamicela
Contextual language models have been trained on Classical languages,
including Ancient Greek and Latin, for tasks such as lemmatization,
morphological tagging, part of speech tagging, authorship attribution, and
detection of scribal errors. However, high-quality sentence embedding models
for these historical languages are significantly more difficult to achieve due
to the lack of training data. In this work, we use a multilingual knowledge
distillation approach to train BERT models to produce sentence embeddings for
Ancient Greek text. The state-of-the-art sentence embedding approaches for
high-resource languages use massive datasets, but our distillation approach
allows our Ancient Greek models to inherit the properties of these models while
using a relatively small amount of translated sentence data. We build a
parallel sentence dataset using a sentence-embedding alignment method to align
Ancient Greek documents with English translations, and use this dataset to
train our models. We evaluate our models on translation search, semantic
similarity, and semantic retrieval tasks and investigate translation bias. We
make our training and evaluation datasets freely available at
https://github.com/kevinkrahn/ancient-greek-datasets .