Adaptive Self-training Framework for Fine-grained Scene Graph Generation
Kibum Kim Kanghoon Yoon Yeonjun In Jinyoung Moon Donghyun Kim Chanyoung Park
Scene graph generation (SGG) models have suffered from inherent problems
regarding the benchmark datasets such as the long-tailed predicate distribution
and missing annotation problems. In this work, we aim to alleviate the
long-tailed problem of SGG by utilizing unannotated triplets. To this end, we
introduce a Self-Training framework for SGG (ST-SGG) that assigns pseudo-labels
for unannotated triplets based on which the SGG models are trained. While there
has been significant progress in self-training for image recognition, designing
a self-training framework for the SGG task is more challenging due to its
inherent nature such as the semantic ambiguity and the long-tailed distribution
of predicate classes. Hence, we propose a novel pseudo-labeling technique for
SGG, called Class-specific Adaptive Thresholding with Momentum (CATM), which is
a model-agnostic framework that can be applied to any existing SGG models.
Furthermore, we devise a graph structure learner (GSL) that is beneficial when
adopting our proposed self-training framework to the state-of-the-art
message-passing neural network (MPNN)-based SGG models. Our extensive
experiments verify the effectiveness of ST-SGG on various SGG models,
particularly in enhancing the performance on fine-grained predicate classes.