Improving Scene Graph Generation with Relation Words' Debiasing in
  Vision-Language Models
Yuxuan Wang Xiaoyuan Liu
Scene Graph Generation (SGG) provides basic language representation of visual
scenes, requiring models to grasp complex and diverse semantics between various
objects. However, this complexity and diversity in SGG also leads to
underrepresentation, where part of test triplets are rare or even unseen during
training, resulting in imprecise predictions. To tackle this, we propose using
the SGG models with pretrained vision-language models (VLMs) to enhance
representation. However, due to the gap between the pretraining and SGG,
directly ensembling the pretrained VLMs leads to severe biases across relation
words. Thus, we introduce LM Estimation to approximate the words' distribution
underlies in the pretraining language sets, and then use the distribution for
debiasing. After that, we ensemble VLMs with SGG models to enhance
representation. Considering that each model may represent better at different
samples, we use a certainty-aware indicator to score each sample and
dynamically adjust the ensemble weights. Our method effectively addresses the
words biases, enhances SGG's representation, and achieve markable performance
enhancements. It is training-free and integrates well with existing SGG models.