FootGPT : A Large Language Model Development Experiment on a Minimal
  Setting
Eren Unlu
With recent empirical observations, it has been argued that the most
significant aspect of developing accurate language models may be the proper
dataset content and training strategy compared to the number of neural
parameters, training duration or dataset size. Following this argument, we
opted to fine tune a one billion parameter size trained general purpose causal
language model with a dataset curated on team statistics of the Italian
football league first ten game weeks, using low rank adaptation. The limited
training dataset was compiled based on a framework where a powerful commercial
large language model provides distilled paragraphs and question answer pairs as
intended. The training duration was kept relatively short to provide a basis
for our minimal setting exploration. We share our key observations on the
process related to developing a specific purpose language model which is
intended to interpret soccer data with constrained resources in this article.