How Important is Domain Specificity in Language Models and Instruction
  Finetuning for Biomedical Relation Extraction?
Aviv Brokman Ramakanth Kavuluru
Cutting edge techniques developed in the general NLP domain are often
subsequently applied to the high-value, data-rich biomedical domain. The past
few years have seen generative language models (LMs), instruction finetuning,
and few-shot learning become foci of NLP research. As such, generative LMs
pretrained on biomedical corpora have proliferated and biomedical instruction
finetuning has been attempted as well, all with the hope that domain
specificity improves performance on downstream tasks. Given the nontrivial
effort in training such models, we investigate what, if any, benefits they have
in the key biomedical NLP task of relation extraction. Specifically, we address
two questions: (1) Do LMs trained on biomedical corpora outperform those
trained on general domain corpora? (2) Do models instruction finetuned on
biomedical datasets outperform those finetuned on assorted datasets or those
simply pretrained? We tackle these questions using existing LMs, testing across
four datasets. In a surprising result, general-domain models typically
outperformed biomedical-domain models. However, biomedical instruction
finetuning improved performance to a similar degree as general instruction
finetuning, despite having orders of magnitude fewer instructions. Our findings
suggest it may be more fruitful to focus research effort on larger-scale
biomedical instruction finetuning of general LMs over building domain-specific
biomedical LMs