UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for
  Biomedical Entity Recognition
Aidan Mannion Thierry Chevalier Didier Schwab Lorraine Geouriot
Pre-trained transformer language models (LMs) have in recent years become the
dominant paradigm in applied NLP. These models have achieved state-of-the-art
performance on tasks such as information extraction, question answering,
sentiment analysis, document classification and many others. In the biomedical
domain, significant progress has been made in adapting this paradigm to NLP
tasks that require the integration of domain-specific knowledge as well as
statistical modelling of language. In particular, research in this area has
focused on the question of how best to construct LMs that take into account not
only the patterns of token distribution in medical text, but also the wealth of
structured information contained in terminology resources such as the UMLS.
This work contributes a data-centric paradigm for enriching the language
representations of biomedical transformer-encoder LMs by extracting text
sequences from the UMLS. This allows for graph-based learning objectives to be
combined with masked-language pre-training. Preliminary results from
experiments in the extension of pre-trained LMs as well as training from
scratch show that this framework improves downstream performance on multiple
biomedical and clinical Named Entity Recognition (NER) tasks.