The Case Records of ChatGPT: Language Models and Complex Clinical
  Questions
Timothy Poterucha Pierre Elias Christopher M. Haggerty
Background: Artificial intelligence language models have shown promise in
various applications, including assisting with clinical decision-making as
demonstrated by strong performance of large language models on medical
licensure exams. However, their ability to solve complex, open-ended cases,
which may be representative of clinical practice, remains unexplored. Methods:
In this study, the accuracy of large language AI models GPT4 and GPT3.5 in
diagnosing complex clinical cases was investigated using published Case Records
of the Massachusetts General Hospital. A total of 50 cases requiring a
diagnosis and diagnostic test published from January 1, 2022 to April 16, 2022
were identified. For each case, models were given a prompt requesting the top
three specific diagnoses and associated diagnostic tests, followed by case
text, labs, and figure legends. Model outputs were assessed in comparison to
the final clinical diagnosis and whether the model-predicted test would result
in a correct diagnosis. Results: GPT4 and GPT3.5 accurately provided the
correct diagnosis in 26% and 22% of cases in one attempt, and 46% and 42%
within three attempts, respectively. GPT4 and GPT3.5 provided a correct
essential diagnostic test in 28% and 24% of cases in one attempt, and 44% and
50% within three attempts, respectively. No significant differences were found
between the two models, and multiple trials with identical prompts using the
GPT3.5 model provided similar results. Conclusions: In summary, these models
demonstrate potential usefulness in generating differential diagnoses but
remain limited in their ability to provide a single unifying diagnosis in
complex, open-ended cases. Future research should focus on evaluating model
performance in larger datasets of open-ended clinical challenges and exploring
potential human-AI collaboration strategies to enhance clinical
decision-making.