Improving Vision-and-Language Reasoning via Spatial Relations Modeling
Cheng Yang Rui Xu Ye Guo Peixiang Huang Yiru Chen Wenkui Ding Zhongyuan Wang Hong Zhou
Visual commonsense reasoning (VCR) is a challenging multi-modal task, which
requires high-level cognition and commonsense reasoning ability about the real
world. In recent years, large-scale pre-training approaches have been developed
and promoted the state-of-the-art performance of VCR. However, the existing
approaches almost employ the BERT-like objectives to learn multi-modal
representations. These objectives motivated from the text-domain are
insufficient for the excavation on the complex scenario of visual modality.
Most importantly, the spatial distribution of the visual objects is basically
neglected. To address the above issue, we propose to construct the spatial
relation graph based on the given visual scenario. Further, we design two
pre-training tasks named object position regression (OPR) and spatial relation
classification (SRC) to learn to reconstruct the spatial relation graph
respectively. Quantitative analysis suggests that the proposed method can guide
the representations to maintain more spatial context and facilitate the
attention on the essential visual regions for reasoning. We achieve the
state-of-the-art results on VCR and two other vision-and-language reasoning
tasks VQA, and NLVR.