EyeGPT: Ophthalmic Assistant with Large Language Models
Xiaolan Chen Ziwei Zhao Weiyi Zhang Pusheng Xu Le Gao Mingpu Xu Yue Wu Yinwen Li Danli Shi Mingguang He
Artificial intelligence (AI) has gained significant attention in healthcare
consultation due to its potential to improve clinical workflow and enhance
medical communication. However, owing to the complex nature of medical
information, large language models (LLM) trained with general world knowledge
might not possess the capability to tackle medical-related tasks at an expert
level. Here, we introduce EyeGPT, a specialized LLM designed specifically for
ophthalmology, using three optimization strategies including role-playing,
finetuning, and retrieval-augmented generation. In particular, we proposed a
comprehensive evaluation framework that encompasses a diverse dataset, covering
various subspecialties of ophthalmology, different users, and diverse inquiry
intents. Moreover, we considered multiple evaluation metrics, including
accuracy, understandability, trustworthiness, empathy, and the proportion of
hallucinations. By assessing the performance of different EyeGPT variants, we
identify the most effective one, which exhibits comparable levels of
understandability, trustworthiness, and empathy to human ophthalmologists (all
Ps>0.05). Overall, ur study provides valuable insights for future research,
facilitating comprehensive comparisons and evaluations of different strategies
for developing specialized LLMs in ophthalmology. The potential benefits
include enhancing the patient experience in eye care and optimizing
ophthalmologists' services.