Boosting Human-Object Interaction Detection with Text-to-Image Diffusion
  Model
Jie Yang Bingliang Li Fengyu Yang Ailing Zeng Lei Zhang Ruimao Zhang
This paper investigates the problem of the current HOI detection methods and
introduces DiffHOI, a novel HOI detection scheme grounded on a pre-trained
text-image diffusion model, which enhances the detector's performance via
improved data diversity and HOI representation. We demonstrate that the
internal representation space of a frozen text-to-image diffusion model is
highly relevant to verb concepts and their corresponding context. Accordingly,
we propose an adapter-style tuning method to extract the various semantic
associated representation from a frozen diffusion model and CLIP model to
enhance the human and object representations from the pre-trained detector,
further reducing the ambiguity in interaction prediction. Moreover, to fill in
the gaps of HOI datasets, we propose SynHOI, a class-balance, large-scale, and
high-diversity synthetic dataset containing over 140K HOI images with fully
triplet annotations. It is built using an automatic and scalable pipeline
designed to scale up the generation of diverse and high-precision HOI-annotated
data. SynHOI could effectively relieve the long-tail issue in existing datasets
and facilitate learning interaction representations. Extensive experiments
demonstrate that DiffHOI significantly outperforms the state-of-the-art in
regular detection (i.e., 41.50 mAP) and zero-shot detection. Furthermore,
SynHOI can improve the performance of model-agnostic and backbone-agnostic HOI
detection, particularly exhibiting an outstanding 11.55% mAP improvement in
rare classes.