Informative Scene Graph Generation via Debiasing
Lianli Gao Xinyu Lyu Yuyu Guo Yuxuan Hu Yuan-Fang Li Lu Xu Heng Tao Shen Jingkuan Song
Scene graph generation aims to detect visual relationship triplets, (subject,
predicate, object). Due to biases in data, current models tend to predict
common predicates, e.g. "on" and "at", instead of informative ones, e.g.
"standing on" and "looking at". This tendency results in the loss of precise
information and overall performance. If a model only uses "stone on road"
rather than "stone blocking road" to describe an image, it may be a grave
misunderstanding. We argue that this phenomenon is caused by two imbalances:
semantic space level imbalance and training sample level imbalance. For this
problem, we propose DB-SGG, an effective framework based on debiasing but not
the conventional distribution fitting. It integrates two components: Semantic
Debiasing (SD) and Balanced Predicate Learning (BPL), for these imbalances. SD
utilizes a confusion matrix and a bipartite graph to construct predicate
relationships. BPL adopts a random undersampling strategy and an ambiguity
removing strategy to focus on informative predicates. Benefiting from the
model-agnostic process, our method can be easily applied to SGG models and
outperforms Transformer by 136.3%, 119.5%, and 122.6% on mR@20 at three SGG
sub-tasks on the SGG-VG dataset. Our method is further verified on another
complex SGG dataset (SGG-GQA) and two downstream tasks (sentence-to-graph
retrieval and image captioning).