Object Attribute Matters in Visual Question Answering
Peize Li Qingyi Si Peng Fu Zheng Lin Yan Wang
Visual question answering is a multimodal task that requires the joint
comprehension of visual and textual information. However, integrating visual
and textual semantics solely through attention layers is insufficient to
comprehensively understand and align information from both modalities.
Intuitively, object attributes can naturally serve as a bridge to unify them,
which has been overlooked in previous research. In this paper, we propose a
novel VQA approach from the perspective of utilizing object attribute, aiming
to achieve better object-level visual-language alignment and multimodal scene
understanding. Specifically, we design an attribute fusion module and a
contrastive knowledge distillation module. The attribute fusion module
constructs a multimodal graph neural network to fuse attributes and visual
features through message passing. The enhanced object-level visual features
contribute to solving fine-grained problem like counting-question. The better
object-level visual-language alignment aids in understanding multimodal scenes,
thereby improving the model's robustness. Furthermore, to augment scene
understanding and the out-of-distribution performance, the contrastive
knowledge distillation module introduces a series of implicit knowledge. We
distill knowledge into attributes through contrastive loss, which further
strengthens the representation learning of attribute features and facilitates
visual-linguistic alignment. Intensive experiments on six datasets, COCO-QA,
VQAv2, VQA-CPv2, VQA-CPv1, VQAvs and TDIUC, show the superiority of the
proposed method.