RepSGG: Novel Representations of Entities and Relationships for Scene
  Graph Generation
Hengyue Liu Bir Bhanu
Scene Graph Generation (SGG) has achieved significant progress recently.
However, most previous works rely heavily on fixed-size entity representations
based on bounding box proposals, anchors, or learnable queries. As each
representation's cardinality has different trade-offs between performance and
computation overhead, extracting highly representative features efficiently and
dynamically is both challenging and crucial for SGG. In this work, a novel
architecture called RepSGG is proposed to address the aforementioned
challenges, formulating a subject as queries, an object as keys, and their
relationship as the maximum attention weight between pairwise queries and keys.
With more fine-grained and flexible representation power for entities and
relationships, RepSGG learns to sample semantically discriminative and
representative points for relationship inference. Moreover, the long-tailed
distribution also poses a significant challenge for generalization of SGG. A
run-time performance-guided logit adjustment (PGLA) strategy is proposed such
that the relationship logits are modified via affine transformations based on
run-time performance during training. This strategy encourages a more balanced
performance between dominant and rare classes. Experimental results show that
RepSGG achieves the state-of-the-art or comparable performance on the Visual
Genome and Open Images V6 datasets with fast inference speed, demonstrating the
efficacy and efficiency of the proposed methods.