Few-shot medical image classification with simple shape and texture text
  descriptors using vision-language models
Michal Byra Muhammad Febrian Rachmadi Henrik Skibbe
In this work, we investigate the usefulness of vision-language models (VLMs)
and large language models for binary few-shot classification of medical images.
We utilize the GPT-4 model to generate text descriptors that encapsulate the
shape and texture characteristics of objects in medical images. Subsequently,
these GPT-4 generated descriptors, alongside VLMs pre-trained on natural
images, are employed to classify chest X-rays and breast ultrasound images. Our
results indicate that few-shot classification of medical images using VLMs and
GPT-4 generated descriptors is a viable approach. However, accurate
classification requires to exclude certain descriptors from the calculations of
the classification scores. Moreover, we assess the ability of VLMs to evaluate
shape features in breast mass ultrasound images. We further investigate the
degree of variability among the sets of text descriptors produced by GPT-4. Our
work provides several important insights about the application of VLMs for
medical image analysis.