Diversifying Knowledge Enhancement of Biomedical Language Models using
  Adapter Modules and Knowledge Graphs
Juraj Vladika Alexander Fichtl Florian Matthes
Recent advances in natural language processing (NLP) owe their success to
pre-training language models on large amounts of unstructured data. Still,
there is an increasing effort to combine the unstructured nature of LMs with
structured knowledge and reasoning. Particularly in the rapidly evolving field
of biomedical NLP, knowledge-enhanced language models (KELMs) have emerged as
promising tools to bridge the gap between large language models and
domain-specific knowledge, considering the available biomedical knowledge
graphs (KGs) curated by experts over the decades. In this paper, we develop an
approach that uses lightweight adapter modules to inject structured biomedical
knowledge into pre-trained language models (PLMs). We use two large KGs, the
biomedical knowledge system UMLS and the novel biochemical ontology OntoChem,
with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. The approach
includes partitioning knowledge graphs into smaller subgraphs, fine-tuning
adapter modules for each subgraph, and combining the knowledge in a fusion
layer. We test the performance on three downstream tasks: document
classification,question answering, and natural language inference. We show that
our methodology leads to performance improvements in several instances while
keeping requirements in computing power low. Finally, we provide a detailed
interpretation of the results and report valuable insights for future work.