RaDialog: A Large Vision-Language Model for Radiology Report Generation
  and Conversational Assistance
Chantal Pellegrini Ege Özsoy Benjamin Busam Nassir Navab Matthias Keicher
Conversational AI tools that can generate and discuss clinically correct
radiology reports for a given medical image have the potential to transform
radiology. Such a human-in-the-loop radiology assistant could facilitate a
collaborative diagnostic process, thus saving time and improving the quality of
reports. Towards this goal, we introduce RaDialog, the first thoroughly
evaluated and publicly available large vision-language model for radiology
report generation and interactive dialog. RaDialog effectively integrates
visual image features and structured pathology findings with a large language
model (LLM) while simultaneously adapting it to a specialized domain using
parameter-efficient fine-tuning. To keep the conversational abilities of the
underlying LLM, we propose a comprehensive, semi-automatically labeled,
image-grounded instruct dataset for chest X-ray radiology tasks. By training
with this dataset, our method achieves state-of-the-art clinical correctness in
report generation and shows impressive abilities in interactive tasks such as
correcting reports and answering questions, serving as a foundational step
toward clinical dialog systems. Our code is available on github:
https://github.com/ChantalMP/RaDialog.