Pretrained Generative Language Models as General Learning Frameworks for
  Sequence-Based Tasks
Ben Fauber
We propose that small pretrained foundational generative language models with
millions of parameters can be utilized as a general learning framework for
sequence-based tasks. Our proposal overcomes the computational resource, skill
set, and timeline challenges associated with training neural networks and
language models from scratch. Further, our approach focuses on creating small
and highly specialized models that can accurately execute a challenging task of
which the base model is incapable of performing. We demonstrate that 125M,
350M, and 1.3B parameter pretrained foundational language models can be
instruction fine-tuned with 10,000-to-1,000,000 instruction examples to achieve
near state-of-the-art results on challenging cheminformatics tasks. We also
demonstrate the role of successive language model fine-tuning epochs on
improved outcomes, as well as the importance of both data formatting and
pretrained foundational language model selection for instruction fine-tuning
success.