Relation Extraction in underexplored biomedical domains: A
  diversity-optimised sampling and synthetic data generation approach
Maxime Delmas Magdalena Wysocka André Freitas
The sparsity of labelled data is an obstacle to the development of Relation
Extraction models and the completion of databases in various biomedical areas.
While being of high interest in drug-discovery, the natural-products
literature, reporting the identification of potential bioactive compounds from
organisms, is a concrete example of such an overlooked topic. To mark the start
of this new task, we created the first curated evaluation dataset and extracted
literature items from the LOTUS database to build training sets. To this end,
we developed a new sampler inspired by diversity metrics in ecology, named
Greedy Maximum Entropy sampler, or GME-sampler
(https://github.com/idiap/gme-sampler). The strategic optimization of both
balance and diversity of the selected items in the evaluation set is important
given the resource-intensive nature of manual curation. After quantifying the
noise in the training set, in the form of discrepancies between the input
abstracts text and the expected output labels, we explored different strategies
accordingly. Framing the task as an end-to-end Relation Extraction, we
evaluated the performance of standard fine-tuning as a generative task and
few-shot learning with open Large Language Models (LLaMA 7B-65B). In addition
to their evaluation in few-shot settings, we explore the potential of open
Large Language Models (Vicuna-13B) as synthetic data generator and propose a
new workflow for this purpose. All evaluated models exhibited substantial
improvements when fine-tuned on synthetic abstracts rather than the original
noisy data. We provide our best performing (f1-score=59.0) BioGPT-Large model
for end-to-end RE of natural-products relationships along with all the
generated synthetic data and the evaluation dataset. See more details at
https://github.com/idiap/abroad-re.