Towards Unseen Triples: Effective Text-Image-joint Learning for Scene
  Graph Generation
Qianji Di Wenxi Ma Zhongang Qi Tianxiang Hou Ying Shan Hanzi Wang
Scene Graph Generation (SGG) aims to structurally and comprehensively
represent objects and their connections in images, it can significantly benefit
scene understanding and other related downstream tasks. Existing SGG models
often struggle to solve the long-tailed problem caused by biased datasets.
However, even if these models can fit specific datasets better, it may be hard
for them to resolve the unseen triples which are not included in the training
set. Most methods tend to feed a whole triple and learn the overall features
based on statistical machine learning. Such models have difficulty predicting
unseen triples because the objects and predicates in the training set are
combined differently as novel triples in the test set. In this work, we propose
a Text-Image-joint Scene Graph Generation (TISGG) model to resolve the unseen
triples and improve the generalisation capability of the SGG models. We propose
a Joint Fearture Learning (JFL) module and a Factual Knowledge based Refinement
(FKR) module to learn object and predicate categories separately at the feature
level and align them with corresponding visual features so that the model is no
longer limited to triples matching. Besides, since we observe the long-tailed
problem also affects the generalization ability, we design a novel balanced
learning strategy, including a Charater Guided Sampling (CGS) and an
Informative Re-weighting (IR) module, to provide tailor-made learning methods
for each predicate according to their characters. Extensive experiments show
that our model achieves state-of-the-art performance. In more detail, TISGG
boosts the performances by 11.7% of zR@20(zero-shot recall) on the PredCls
sub-task on the Visual Genome dataset.