Hierarchical Pretraining for Biomedical Term Embeddings
Bryan Cai Sihang Zeng Yucong Lin Zheng Yuan Doudou Zhou Lu Tian
Electronic health records (EHR) contain narrative notes that provide
extensive details on the medical condition and management of patients. Natural
language processing (NLP) of clinical notes can use observed frequencies of
clinical terms as predictive features for downstream applications such as
clinical decision making and patient trajectory prediction. However, due to the
vast number of highly similar and related clinical concepts, a more effective
modeling strategy is to represent clinical terms as semantic embeddings via
representation learning and use the low dimensional embeddings as feature
vectors for predictive modeling. To achieve efficient representation,
fine-tuning pretrained language models with biomedical knowledge graphs may
generate better embeddings for biomedical terms than those from standard
language models alone. These embeddings can effectively discriminate synonymous
pairs of from those that are unrelated. However, they often fail to capture
different degrees of similarity or relatedness for concepts that are
hierarchical in nature. To overcome this limitation, we propose HiPrBERT, a
novel biomedical term representation model trained on additionally complied
data that contains hierarchical structures for various biomedical terms. We
modify an existing contrastive loss function to extract information from these
hierarchies. Our numerical experiments demonstrate that HiPrBERT effectively
learns the pair-wise distance from hierarchical information, resulting in a
substantially more informative embeddings for further biomedical applications