SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based
  Question Answering
Bruno Souza Marius Aasan Helio Pedrini Adín Ramírez Rivera
The intersection of vision and language is of major interest due to the
increased focus on seamless integration between recognition and reasoning.
Scene graphs (SGs) have emerged as a useful tool for multimodal image analysis,
showing impressive performance in tasks such as Visual Question Answering
(VQA). In this work, we demonstrate that despite the effectiveness of scene
graphs in VQA tasks, current methods that utilize idealized annotated scene
graphs struggle to generalize when using predicted scene graphs extracted from
images. To address this issue, we introduce the SelfGraphVQA framework. Our
approach extracts a scene graph from an input image using a pre-trained scene
graph generator and employs semantically-preserving augmentation with
self-supervised techniques. This method improves the utilization of graph
representations in VQA tasks by circumventing the need for costly and
potentially biased annotated data. By creating alternative views of the
extracted graphs through image augmentations, we can learn joint embeddings by
optimizing the informational content in their representations using an
un-normalized contrastive approach. As we work with SGs, we experiment with
three distinct maximization strategies: node-wise, graph-wise, and
permutation-equivariant regularization. We empirically showcase the
effectiveness of the extracted scene graph for VQA and demonstrate that these
approaches enhance overall performance by highlighting the significance of
visual information. This offers a more practical solution for VQA tasks that
rely on SGs for complex reasoning questions.