Preserving the knowledge of long clinical texts using aggregated
  ensembles of large language models
Mohammad Junayed Hasan Suhra Noor Mohammad Ashrafuzzaman Khan
Clinical texts, such as admission notes, discharge summaries, and progress
notes, contain rich and valuable information that can be used for various
clinical outcome prediction tasks. However, applying large language models,
such as BERT-based models, to clinical texts poses two major challenges: the
limitation of input length and the diversity of data sources. This paper
proposes a novel method to preserve the knowledge of long clinical texts using
aggregated ensembles of large language models. Unlike previous studies which
use model ensembling or text aggregation methods separately, we combine
ensemble learning with text aggregation and train multiple large language
models on two clinical outcome tasks: mortality prediction and length of stay
prediction. We show that our method can achieve better results than baselines,
ensembling, and aggregation individually, and can improve the performance of
large language models while handling long inputs and diverse datasets. We
conduct extensive experiments on the admission notes from the MIMIC-III
clinical database by combining multiple unstructured and high-dimensional
datasets, demonstrating our method's effectiveness and superiority over
existing approaches. We also provide a comprehensive analysis and discussion of
our results, highlighting our method's applications and limitations for future
research in the domain of clinical healthcare. The results and analysis of this
study is supportive of our method assisting in clinical healthcare systems by
enabling clinical decision-making with robust performance overcoming the
challenges of long text inputs and varied datasets.