Use GPT-J Prompt Generation with RoBERTa for NER Models on Diagnosis
  Extraction of Periodontal Diagnosis from Electronic Dental Records
Yao-Shun Chuang Xiaoqian Jiang Chun-Teh Lee Ryan Brandon Duong Tran Oluwabunmi Tokede Muhammad F. Walji
This study explored the usability of prompt generation on named entity
recognition (NER) tasks and the performance in different settings of the
prompt. The prompt generation by GPT-J models was utilized to directly test the
gold standard as well as to generate the seed and further fed to the RoBERTa
model with the spaCy package. In the direct test, a lower ratio of negative
examples with higher numbers of examples in prompt achieved the best results
with a F1 score of 0.72. The performance revealed consistency, 0.92-0.97 in the
F1 score, in all settings after training with the RoBERTa model. The study
highlighted the importance of seed quality rather than quantity in feeding NER
models. This research reports on an efficient and accurate way to mine clinical
notes for periodontal diagnoses, allowing researchers to easily and quickly
build a NER model with the prompt generation approach.