Evaluating multiple large language models in pediatric ophthalmology
Jason Holmes Rui Peng Yiwei Li Jinyu Hu Zhengliang Liu Zihao Wu Huan Zhao Xi Jiang Wei Liu Hong Wei Jie Zou Tianming Liu Yi Shao
IMPORTANCE The response effectiveness of different large language models
(LLMs) and various individuals, including medical students, graduate students,
and practicing physicians, in pediatric ophthalmology consultations, has not
been clearly established yet. OBJECTIVE Design a 100-question exam based on
pediatric ophthalmology to evaluate the performance of LLMs in highly
specialized scenarios and compare them with the performance of medical students
and physicians at different levels. DESIGN, SETTING, AND PARTICIPANTS This
survey study assessed three LLMs, namely ChatGPT (GPT-3.5), GPT-4, and PaLM2,
were assessed alongside three human cohorts: medical students, postgraduate
students, and attending physicians, in their ability to answer questions
related to pediatric ophthalmology. It was conducted by administering
questionnaires in the form of test papers through the LLM network interface,
with the valuable participation of volunteers. MAIN OUTCOMES AND MEASURES Mean
scores of LLM and humans on 100 multiple-choice questions, as well as the
answer stability, correlation, and response confidence of each LLM. RESULTS
GPT-4 performed comparably to attending physicians, while ChatGPT (GPT-3.5) and
PaLM2 outperformed medical students but slightly trailed behind postgraduate
students. Furthermore, GPT-4 exhibited greater stability and confidence when
responding to inquiries compared to ChatGPT (GPT-3.5) and PaLM2. CONCLUSIONS
AND RELEVANCE Our results underscore the potential for LLMs to provide medical
assistance in pediatric ophthalmology and suggest significant capacity to guide
the education of medical students.