Low-resource classification of mobility functioning information in
  clinical sentences using large language models
Tuan Dung Le Thanh Duong Thanh Thieu
Objective: Function is increasingly recognized as an important indicator of
whole-person health. This study evaluates the ability of publicly available
large language models (LLMs) to accurately identify the presence of functioning
information from clinical notes. We explore various strategies to improve the
performance on this task. Materials and Methods: We collect a balanced binary
classification dataset of 1000 sentences from the Mobility NER dataset, which
was curated from n2c2 clinical notes. For evaluation, we construct zero-shot
and few-shot prompts to query the LLMs whether a given sentence contains
mobility functioning information. Two sampling techniques, random sampling and
k-nearest neighbor (kNN)-based sampling, are used to select the few-shot
examples. Furthermore, we apply a parameter-efficient prompt-based fine-tuning
method to the LLMs and evaluate their performance under various training
settings. Results: Flan-T5-xxl outperforms all other models in both zero-shot
and few-shot settings, achieving a F1 score of 0.865 with a single
demonstrative example selected by kNN sampling. In prompt-based fine-tuning
experiments, this foundation model also demonstrates superior performance
across all low-resource settings, particularly achieving an impressive F1 score
of 0.922 using the full training dataset. The smaller model, Flan-T5-xl,
requires fine-tuning with only 2.3M additional parameters to achieve comparable
performance to the fully fine-tuned Gatortron-base model, both surpassing 0.9
F1 score. Conclusion: Open-source instruction-tuned LLMs demonstrate impressive
in-context learning capability in the mobility functioning classification task.
The performance of these models can be further improved by continuing
fine-tuning on a task-specific dataset.