Enhancing Scene Graph Generation with Hierarchical Relationships and
  Commonsense Knowledge
Bowen Jiang Zhijun Zhuang Camillo Jose Taylor
This work presents an enhanced approach to generating scene graphs by
incorporating a relationship hierarchy and commonsense knowledge. Specifically,
we propose a Bayesian classification head that exploits an informative
hierarchical structure. It jointly predicts the super-category or type of
relationship between the two objects, along with the detailed relationship
under each super-category. We design a commonsense validation pipeline that
uses a large language model to critique the results from the scene graph
prediction system and then use that feedback to enhance the model performance.
The system requires no external large language model assistance at test time,
making it more convenient for practical applications. Experiments on the Visual
Genome and the OpenImage V6 datasets demonstrate that harnessing hierarchical
relationships enhances the model performance by a large margin. The proposed
Bayesian head can also be incorporated as a portable module in existing scene
graph generation algorithms to improve their results. In addition, the
commonsense validation enables the model to generate an extensive set of
reasonable predictions beyond dataset annotations.