RadLing: Towards Efficient Radiology Report Understanding
Rikhiya Ghosh Sanjeev Kumar Karn Manuela Daniela Danu Larisa Micu Ramya Vunikili Oladimeji Farri
Most natural language tasks in the radiology domain use language models
pre-trained on biomedical corpus. There are few pretrained language models
trained specifically for radiology, and fewer still that have been trained in a
low data setting and gone on to produce comparable results in fine-tuning
tasks. We present RadLing, a continuously pretrained language model using
Electra-small (Clark et al., 2020) architecture, trained using over 500K
radiology reports, that can compete with state-of-the-art results for fine
tuning tasks in radiology domain. Our main contribution in this paper is
knowledge-aware masking which is a taxonomic knowledge-assisted pretraining
task that dynamically masks tokens to inject knowledge during pretraining. In
addition, we also introduce an knowledge base-aided vocabulary extension to
adapt the general tokenization vocabulary to radiology domain.