Performance Assessment of ChatGPT vs Bard in Detecting Alzheimer's
  Dementia
Balamurali B T Jer-Ming Chen
Large language models (LLMs) find increasing applications in many fields.
Here, three LLM chatbots (ChatGPT-3.5, ChatGPT-4 and Bard) are assessed - in
their current form, as publicly available - for their ability to recognize
Alzheimer's Dementia (AD) and Cognitively Normal (CN) individuals using textual
input derived from spontaneous speech recordings. Zero-shot learning approach
is used at two levels of independent queries, with the second query
(chain-of-thought prompting) eliciting more detailed than the first. Each LLM
chatbot's performance is evaluated on the prediction generated in terms of
accuracy, sensitivity, specificity, precision and F1 score. LLM chatbots
generated three-class outcome ("AD", "CN", or "Unsure"). When positively
identifying AD, Bard produced highest true-positives (89% recall) and highest
F1 score (71%), but tended to misidentify CN as AD, with high confidence (low
"Unsure" rates); for positively identifying CN, GPT-4 resulted in the highest
true-negatives at 56% and highest F1 score (62%), adopting a diplomatic stance
(moderate "Unsure" rates). Overall, three LLM chatbots identify AD vs CN
surpassing chance-levels but do not currently satisfy clinical application.