Comparative Analysis of Drug-GPT and ChatGPT LLMs for Healthcare
  Insights: Evaluating Accuracy and Relevance in Patient and HCP Contexts
Giorgos Lysandrou Roma English Owen Kirsty Mursec Grant Le Brun Elizabeth A. L. Fairley
This study presents a comparative analysis of three Generative Pre-trained
Transformer (GPT) solutions in a question and answer (Q&A) setting: Drug-GPT 3,
Drug-GPT 4, and ChatGPT, in the context of healthcare applications. The
objective is to determine which model delivers the most accurate and relevant
information in response to prompts related to patient experiences with atopic
dermatitis (AD) and healthcare professional (HCP) discussions about diabetes.
The results demonstrate that while all three models are capable of generating
relevant and accurate responses, Drug-GPT 3 and Drug-GPT 4, which are supported
by curated datasets of patient and HCP social media and message board posts,
provide more targeted and in-depth insights. ChatGPT, a more general-purpose
model, generates broader and more general responses, which may be valuable for
readers seeking a high-level understanding of the topics but may lack the depth
and personal insights found in the answers generated by the specialized
Drug-GPT models. This comparative analysis highlights the importance of
considering the language model's perspective, depth of knowledge, and currency
when evaluating the usefulness of generated information in healthcare
applications.