Efficiency at Scale: Investigating the Performance of Diminutive
  Language Models in Clinical Tasks
Niall Taylor Upamanyu Ghose Omid Rohanian Mohammadmahdi Nouriborji Andrey Kormilitzin David Clifton Alejo Nevado-Holgado
The entry of large language models (LLMs) into research and commercial spaces
has led to a trend of ever-larger models, with initial promises of
generalisability, followed by a widespread desire to downsize and create
specialised models without the need for complete fine-tuning, using Parameter
Efficient Fine-tuning (PEFT) methods. We present an investigation into the
suitability of different PEFT methods to clinical decision-making tasks, across
a range of model sizes, including extremely small models with as few as $25$
million parameters.
  Our analysis shows that the performance of most PEFT approaches varies
significantly from one task to another, with the exception of LoRA, which
maintains relatively high performance across all model sizes and tasks,
typically approaching or matching full fine-tuned performance. The
effectiveness of PEFT methods in the clinical domain is evident, particularly
for specialised models which can operate on low-cost, in-house computing
infrastructure. The advantages of these models, in terms of speed and reduced
training costs, dramatically outweighs any performance gain from large
foundation LLMs. Furthermore, we highlight how domain-specific pre-training
interacts with PEFT methods and model size, and discuss how these factors
interplay to provide the best efficiency-performance trade-off. Full code
available at: tbd.