Benchmarking Large Language Models on Communicative Medical Coaching: a
  Novel System and Dataset
Hengguan Huang Songtao Wang Hongfu Liu Hao Wang Ye Wang
Traditional applications of natural language processing (NLP) in healthcare
have predominantly focused on patient-centered services, enhancing patient
interactions and care delivery, such as through medical dialogue systems.
However, the potential of NLP to benefit inexperienced doctors, particularly in
areas such as communicative medical coaching, remains largely unexplored. We
introduce ``ChatCoach,'' an integrated human-AI cooperative framework. Within
this framework, both a patient agent and a coaching agent collaboratively
support medical learners in practicing their medical communication skills
during consultations. Unlike traditional dialogue systems, ChatCoach provides a
simulated environment where a human doctor can engage in medical dialogue with
a patient agent. Simultaneously, a coaching agent provides real-time feedback
to the doctor. To construct the ChatCoach system, we developed a dataset and
integrated Large Language Models such as ChatGPT and Llama2, aiming to assess
their effectiveness in communicative medical coaching tasks. Our comparative
analysis demonstrates that instruction-tuned Llama2 significantly outperforms
ChatGPT's prompting-based approaches.