M2C: Towards Automatic Multimodal Manga Complement
Hongcheng Guo Boyang Wang Jiaqi Bai Jiaheng Liu Jian Yang Zhoujun Li
Multimodal manga analysis focuses on enhancing manga understanding with
visual and textual features, which has attracted considerable attention from
both natural language processing and computer vision communities. Currently,
most comics are hand-drawn and prone to problems such as missing pages, text
contamination, and aging, resulting in missing comic text content and seriously
hindering human comprehension. In other words, the Multimodal Manga Complement
(M2C) task has not been investigated, which aims to handle the aforementioned
issues by providing a shared semantic space for vision and language
understanding. To this end, we first propose the Multimodal Manga Complement
task by establishing a new M2C benchmark dataset covering two languages. First,
we design a manga argumentation method called MCoT to mine event knowledge in
comics with large language models. Then, an effective baseline FVP-M$^{2}$
using fine-grained visual prompts is proposed to support manga complement.
Extensive experimental results show the effectiveness of FVP-M$^{2}$ method for
Multimodal Mange Complement.