A Benchmark of Domain-Adapted Large Language Models for Generating Brief
  Hospital Course Summaries
Asad Aali Dave Van Veen Yamin Ishraq Arefeen Jason Hom Christian Bluethgen Eduardo Pontes Reis Sergios Gatidis Namuun Clifford Joseph Daws Arash S. Tehrani Jangwon Kim Akshay S. Chaudhari
Brief hospital course (BHC) summaries are common clinical documents generated
by summarizing clinical notes. While large language models (LLMs) depict
remarkable capabilities in automating real-world tasks, their capabilities for
healthcare applications such as BHC synthesis have not been shown. To enable
the adaptation of LLMs for BHC synthesis, we introduce a novel benchmark
consisting of a pre-processed dataset extracted from MIMIC-IV notes,
encapsulating clinical note, and brief hospital course (BHC) pairs. We assess
the performance of two general-purpose LLMs and three healthcare-adapted LLMs
to improve BHC synthesis from clinical notes. Using clinical notes as input for
generating BHCs, we apply prompting-based (using in-context learning) and
fine-tuning-based adaptation strategies to three open-source LLMs
(Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5,
GPT-4). We quantitatively evaluate the performance of these LLMs across varying
context-length inputs using conventional natural language similarity metrics.
We further perform a qualitative study where five diverse clinicians blindly
compare clinician-written BHCs and two LLM-generated BHCs for 30 samples across
metrics of comprehensiveness, conciseness, factual correctness, and fluency.
Overall, we present a new benchmark and pre-processed dataset for using LLMs in
BHC synthesis from clinical notes. We observe high-quality summarization
performance for both in-context proprietary and fine-tuned open-source LLMs
using both quantitative metrics and a qualitative clinical reader study. We
propose our work as a benchmark to motivate future works to adapt and assess
the performance of LLMs in BHC synthesis.