Benchmarking a foundation LLM on its ability to re-label structure names
  in accordance with the AAPM TG-263 report
Jason Holmes Lian Zhang Yuzhen Ding Hongying Feng Zhengliang Liu Tianming Liu William W. Wong Sujay A. Vora Jonathan B. Ashman Wei Liu
Purpose: To introduce the concept of using large language models (LLMs) to
re-label structure names in accordance with the American Association of
Physicists in Medicine (AAPM) Task Group (TG)-263 standard, and to establish a
benchmark for future studies to reference.
  Methods and Materials: The Generative Pre-trained Transformer (GPT)-4
application programming interface (API) was implemented as a Digital Imaging
and Communications in Medicine (DICOM) storage server, which upon receiving a
structure set DICOM file, prompts GPT-4 to re-label the structure names of both
target volumes and normal tissues according to the AAPM TG-263. Three disease
sites, prostate, head and neck, and thorax were selected for evaluation. For
each disease site category, 150 patients were randomly selected for manually
tuning the instructions prompt (in batches of 50) and 50 patients were randomly
selected for evaluation. Structure names that were considered were those that
were most likely to be relevant for studies utilizing structure contours for
many patients.
  Results: The overall re-labeling accuracy of both target volumes and normal
tissues for prostate, head and neck, and thorax cases was 96.0%, 98.5%, and
96.9% respectively. Re-labeling of target volumes was less accurate on average
except for prostate - 100%, 93.1%, and 91.1% respectively.
  Conclusions: Given the accuracy of GPT-4 in re-labeling structure names of
both target volumes and normal tissues as presented in this work, LLMs are
poised to be the preferred method for standardizing structure names in
radiation oncology, especially considering the rapid advancements in LLM
capabilities that are likely to continue.