IQAGPT: Image Quality Assessment with Vision-language and ChatGPT Models
Zhihao Chen Bin Hu Chuang Niu Tao Chen Yuxin Li Hongming Shan Ge Wang
Large language models (LLMs), such as ChatGPT, have demonstrated impressive
capabilities in various tasks and attracted an increasing interest as a natural
language interface across many domains. Recently, large vision-language models
(VLMs) like BLIP-2 and GPT-4 have been intensively investigated, which learn
rich vision-language correlation from image-text pairs. However, despite these
developments, the application of LLMs and VLMs in image quality assessment
(IQA), particularly in medical imaging, remains to be explored, which is
valuable for objective performance evaluation and potential supplement or even
replacement of radiologists' opinions. To this end, this paper introduces
IQAGPT, an innovative image quality assessment system integrating an image
quality captioning VLM with ChatGPT for generating quality scores and textual
reports. First, we build a CT-IQA dataset for training and evaluation,
comprising 1,000 CT slices with diverse quality levels professionally
annotated. To better leverage the capabilities of LLMs, we convert annotated
quality scores into semantically rich text descriptions using a prompt
template. Second, we fine-tune the image quality captioning VLM on the CT-IQA
dataset to generate quality descriptions. The captioning model fuses the image
and text features through cross-modal attention. Third, based on the quality
descriptions, users can talk with ChatGPT to rate image quality scores or
produce a radiological quality report. Our preliminary results demonstrate the
feasibility of assessing image quality with large models. Remarkably, our
IQAGPT outperforms GPT-4 and CLIP-IQA, as well as the multi-task classification
and regression models that solely rely on images.