Improving Scene Graph Generation with Superpixel-Based Interaction
  Learning
Jingyi Wang Can Zhang Jinfa Huang Botao Ren Zhidong Deng
Recent advances in Scene Graph Generation (SGG) typically model the
relationships among entities utilizing box-level features from pre-defined
detectors. We argue that an overlooked problem in SGG is the coarse-grained
interactions between boxes, which inadequately capture contextual semantics for
relationship modeling, practically limiting the development of the field. In
this paper, we take the initiative to explore and propose a generic paradigm
termed Superpixel-based Interaction Learning (SIL) to remedy coarse-grained
interactions at the box level. It allows us to model fine-grained interactions
at the superpixel level in SGG. Specifically, (i) we treat a scene as a set of
points and cluster them into superpixels representing sub-regions of the scene.
(ii) We explore intra-entity and cross-entity interactions among the
superpixels to enrich fine-grained interactions between entities at an earlier
stage. Extensive experiments on two challenging benchmarks (Visual Genome and
Open Image V6) prove that our SIL enables fine-grained interaction at the
superpixel level above previous box-level methods, and significantly
outperforms previous state-of-the-art methods across all metrics. More
encouragingly, the proposed method can be applied to boost the performance of
existing box-level approaches in a plug-and-play fashion. In particular, SIL
brings an average improvement of 2.0% mR (even up to 3.4%) of baselines for the
PredCls task on Visual Genome, which facilitates its integration into any
existing box-level method.