Finetuning an LLM on Contextual Knowledge of Classics for Q&A
Shane Storm Strachan
The open-source publishing of large language models (LLMs) has created many
possibilities for how anyone who understands language and has access to a
computer can interact with significant tools of artificial intelligence,
particularly in the context of learning and knowledge dissemination. However,
the utility of these models in specialized fields like Classics is still
largely unexplored. This project is an attempt to merge the knowledge of
Classics with the capabilities of artificial intelligence by finetuning an LLM
to cater to the specific needs of learners and professionals. The goal of this
project is to develop an LLM that not only reproduces contextual knowledge
accurately but also exhibits a consistent "personality" - and, indeed, has
consistent propriety - to appeal to a diverse audience who possess differing
levels of knowledge. A significant portion of this project was dedicated to
refining the dataset, following the principle of "garbage in, garbage out," to
ensure the model generates relevant, useful, and creative responses when given
a prompt (a statement, question, or single word). After training and
evaluation, my model's ability to handle a vast array of different types of
inputs and prompting exceeded expectations for a 355M parameter model, though
its occasional hallucinations (especially when set with a high temperature),
particularly in its assertions about historical events or its own identity,
make it seem somewhat capricious and more work in the form of continuous
finetuning will be undertaken.