Detecting Any Human-Object Interaction Relationship: Universal HOI
  Detector with Spatial Prompt Learning on Foundation Models
Yichao Cao Qingfei Tang Xiu Su Chen Song Shan You Xiaobo Lu Chang Xu
Human-object interaction (HOI) detection aims to comprehend the intricate
relationships between humans and objects, predicting $<human, action, object>$
triplets, and serving as the foundation for numerous computer vision tasks. The
complexity and diversity of human-object interactions in the real world,
however, pose significant challenges for both annotation and recognition,
particularly in recognizing interactions within an open world context. This
study explores the universal interaction recognition in an open-world setting
through the use of Vision-Language (VL) foundation models and large language
models (LLMs). The proposed method is dubbed as \emph{\textbf{UniHOI}}. We
conduct a deep analysis of the three hierarchical features inherent in visual
HOI detectors and propose a method for high-level relation extraction aimed at
VL foundation models, which we call HO prompt-based learning. Our design
includes an HO Prompt-guided Decoder (HOPD), facilitates the association of
high-level relation representations in the foundation model with various HO
pairs within the image. Furthermore, we utilize a LLM (\emph{i.e.} GPT) for
interaction interpretation, generating a richer linguistic understanding for
complex HOIs. For open-category interaction recognition, our method supports
either of two input types: interaction phrase or interpretive sentence. Our
efficient architecture design and learning methods effectively unleash the
potential of the VL foundation models and LLMs, allowing UniHOI to surpass all
existing methods with a substantial margin, under both supervised and zero-shot
settings. The code and pre-trained weights are available at:
\url{https://github.com/Caoyichao/UniHOI}.