A Multi-Modal Context Reasoning Approach for Conditional Inference on
  Joint Textual and Visual Clues
Yunxin Li Baotian Hu Xinyu Chen Yuxin Ding Lin Ma Min Zhang
Conditional inference on joint textual and visual clues is a multi-modal
reasoning task that textual clues provide prior permutation or external
knowledge, which are complementary with visual content and pivotal to deducing
the correct option. Previous methods utilizing pretrained vision-language
models (VLMs) have achieved impressive performances, yet they show a lack of
multimodal context reasoning capability, especially for text-modal information.
To address this issue, we propose a Multi-modal Context Reasoning approach,
named ModCR. Compared to VLMs performing reasoning via cross modal semantic
alignment, it regards the given textual abstract semantic and objective image
information as the pre-context information and embeds them into the language
model to perform context reasoning. Different from recent vision-aided language
models used in natural language processing, ModCR incorporates the multi-view
semantic alignment information between language and vision by introducing the
learnable alignment prefix between image and text in the pretrained language
model. This makes the language model well-suitable for such multi-modal
reasoning scenario on joint textual and visual clues. We conduct extensive
experiments on two corresponding data sets and experimental results show
significantly improved performance (exact gain by 4.8% on PMR test set)
compared to previous strong baselines. Code Link:
\url{https://github.com/YunxinLi/Multimodal-Context-Reasoning}.