Solving the Right Problem is Key for Translational NLP: A Case Study in
  UMLS Vocabulary Insertion
Bernal Jimenez Gutierrez Yuqing Mao Vinh Nguyen Kin Wah Fung Yu Su Olivier Bodenreider
As the immense opportunities enabled by large language models become more
apparent, NLP systems will be increasingly expected to excel in real-world
settings. However, in many instances, powerful models alone will not yield
translational NLP solutions, especially if the formulated problem is not well
aligned with the real-world task. In this work, we study the case of UMLS
vocabulary insertion, an important real-world task in which hundreds of
thousands of new terms, referred to as atoms, are added to the UMLS, one of the
most comprehensive open-source biomedical knowledge bases. Previous work aimed
to develop an automated NLP system to make this time-consuming, costly, and
error-prone task more efficient. Nevertheless, practical progress in this
direction has been difficult to achieve due to a problem formulation and
evaluation gap between research output and the real-world task. In order to
address this gap, we introduce a new formulation for UMLS vocabulary insertion
which mirrors the real-world task, datasets which faithfully represent it and
several strong baselines we developed through re-purposing existing solutions.
Additionally, we propose an effective rule-enhanced biomedical language model
which enables important new model behavior, outperforms all strong baselines
and provides measurable qualitative improvements to editors who carry out the
UVI task. We hope this case study provides insight into the considerable
importance of problem formulation for the success of translational NLP
solutions.