MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English
  Clinical Queries
Akash Ghosh Arkadeep Acharya Prince Jha Aniket Gaudgaul Rajdeep Majumdar Sriparna Saha Aman Chadha Raghav Jain Setu Sinha Shivani Agarwal
In the healthcare domain, summarizing medical questions posed by patients is
critical for improving doctor-patient interactions and medical decision-making.
Although medical data has grown in complexity and quantity, the current body of
research in this domain has primarily concentrated on text-based methods,
overlooking the integration of visual cues. Also prior works in the area of
medical question summarisation have been limited to the English language. This
work introduces the task of multimodal medical question summarization for
codemixed input in a low-resource setting. To address this gap, we introduce
the Multimodal Medical Codemixed Question Summarization MMCQS dataset, which
combines Hindi-English codemixed medical queries with visual aids. This
integration enriches the representation of a patient's medical condition,
providing a more comprehensive perspective. We also propose a framework named
MedSumm that leverages the power of LLMs and VLMs for this task. By utilizing
our MMCQS dataset, we demonstrate the value of integrating visual information
from images to improve the creation of medically detailed summaries. This
multimodal strategy not only improves healthcare decision-making but also
promotes a deeper comprehension of patient queries, paving the way for future
exploration in personalized and responsive medical care. Our dataset, code, and
pre-trained models will be made publicly available.