CIC: A framework for Culturally-aware Image Captioning
Youngsik Yun Jihie Kim
Image Captioning generates descriptive sentences from images using
Vision-Language Pre-trained models (VLPs) such as BLIP, which has improved
greatly. However, current methods lack the generation of detailed descriptive
captions for the cultural elements depicted in the images, such as the
traditional clothing worn by people from Asian cultural groups. In this paper,
we propose a new framework, \textbf{Culturally-aware Image Captioning (CIC)},
that generates captions and describes cultural elements extracted from cultural
visual elements in images representing cultures. Inspired by methods combining
visual modality and Large Language Models (LLMs) through appropriate prompts,
our framework (1) generates questions based on cultural categories from images,
(2) extracts cultural visual elements from Visual Question Answering (VQA)
using generated questions, and (3) generates culturally-aware captions using
LLMs with the prompts. Our human evaluation conducted on 45 participants from 4
different cultural groups with a high understanding of the corresponding
culture shows that our proposed framework generates more culturally descriptive
captions when compared to the image captioning baseline based on VLPs. Our code
and dataset will be made publicly available upon acceptance.