LOIS: Looking Out of Instance Semantics for Visual Question Answering
Siyu Zhang Yeming Chen Yaoru Sun Fang Wang Haibo Shi Haoran Wang
Visual question answering (VQA) has been intensively studied as a multimodal
task that requires effort in bridging vision and language to infer answers
correctly. Recent attempts have developed various attention-based modules for
solving VQA tasks. However, the performance of model inference is largely
bottlenecked by visual processing for semantics understanding. Most existing
detection methods rely on bounding boxes, remaining a serious challenge for VQA
models to understand the causal nexus of object semantics in images and
correctly infer contextual information. To this end, we propose a finer model
framework without bounding boxes in this work, termed Looking Out of Instance
Semantics (LOIS) to tackle this important issue. LOIS enables more fine-grained
feature descriptions to produce visual facts. Furthermore, to overcome the
label ambiguity caused by instance masks, two types of relation attention
modules: 1) intra-modality and 2) inter-modality, are devised to infer the
correct answers from the different multi-view features. Specifically, we
implement a mutual relation attention module to model sophisticated and deeper
visual semantic relations between instance objects and background information.
In addition, our proposed attention model can further analyze salient image
regions by focusing on important word-related questions. Experimental results
on four benchmark VQA datasets prove that our proposed method has favorable
performance in improving visual reasoning capability.