A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image
  Diagnosis
Jiaxiang Liu Tianxiang Hu Yan Zhang Xiaotang Gai Yang Feng Zuozhu Liu
Zero-shot medical image classification is a critical process in real-world
scenarios where we have limited access to all possible diseases or large-scale
annotated data. It involves computing similarity scores between a query medical
image and possible disease categories to determine the diagnostic result.
Recent advances in pretrained vision-language models (VLMs) such as CLIP have
shown great performance for zero-shot natural image recognition and exhibit
benefits in medical applications. However, an explainable zero-shot medical
image recognition framework with promising performance is yet under
development. In this paper, we propose a novel CLIP-based zero-shot medical
image classification framework supplemented with ChatGPT for explainable
diagnosis, mimicking the diagnostic process performed by human experts. The key
idea is to query large language models (LLMs) with category names to
automatically generate additional cues and knowledge, such as disease symptoms
or descriptions other than a single category name, to help provide more
accurate and explainable diagnosis in CLIP. We further design specific prompts
to enhance the quality of generated texts by ChatGPT that describe visual
medical features. Extensive results on one private dataset and four public
datasets along with detailed analysis demonstrate the effectiveness and
explainability of our training-free zero-shot diagnosis pipeline, corroborating
the great potential of VLMs and LLMs for medical applications.