Benchmarking and Analyzing In-context Learning, Fine-tuning and
  Supervised Learning for Biomedical Knowledge Curation: a focused study on
  chemical entities of biological interest
Emily Groves Minhong Wang Yusuf Abdulle Holger Kunz Jason Hoelscher-Obermaier Ronin Wu Honghan Wu
Automated knowledge curation for biomedical ontologies is key to ensure that
they remain comprehensive, high-quality and up-to-date. In the era of
foundational language models, this study compares and analyzes three NLP
paradigms for curation tasks: in-context learning (ICL), fine-tuning (FT), and
supervised learning (ML). Using the Chemical Entities of Biological Interest
(ChEBI) database as a model ontology, three curation tasks were devised. For
ICL, three prompting strategies were employed with GPT-4, GPT-3.5, BioGPT.
PubmedBERT was chosen for the FT paradigm. For ML, six embedding models were
utilized for training Random Forest and Long-Short Term Memory models. Five
setups were designed to assess ML and FT model performance across different
data availability scenarios.Datasets for curation tasks included: task 1
(620,386), task 2 (611,430), and task 3 (617,381), maintaining a 50:50 positive
versus negative ratio. For ICL models, GPT-4 achieved best accuracy scores of
0.916, 0.766 and 0.874 for tasks 1-3 respectively. In a direct comparison, ML
(trained on ~260,000 triples) outperformed ICL in accuracy across all tasks.
(accuracy differences: +.11, +.22 and +.17). Fine-tuned PubmedBERT performed
similarly to leading ML models in tasks 1 & 2 (F1 differences: -.014 and
+.002), but worse in task 3 (-.048). Simulations revealed performance declines
in both ML and FT models with smaller and higher imbalanced training data.
where ICL (particularly GPT-4) excelled in tasks 1 & 3. GPT-4 excelled in tasks
1 and 3 with less than 6,000 triples, surpassing ML/FT. ICL underperformed
ML/FT in task 2.ICL-augmented foundation models can be good assistants for
knowledge curation with correct prompting, however, not making ML and FT
paradigms obsolete. The latter two require task-specific data to beat ICL. In
such cases, ML relies on small pretrained embeddings, minimizing computational
demands.