Evaluation of AI Chatbots for Patient-Specific EHR Questions
Alaleh Hamidi Kirk Roberts
This paper investigates the use of artificial intelligence chatbots for
patient-specific question answering (QA) from clinical notes using several
large language model (LLM) based systems: ChatGPT (versions 3.5 and 4), Google
Bard, and Claude. We evaluate the accuracy, relevance, comprehensiveness, and
coherence of the answers generated by each model using a 5-point Likert scale
on a set of patient-specific questions.