Generating Human-Centric Visual Cues for Human-Object Interaction
  Detection via Large Vision-Language Models
Yu-Wei Zhan Fan Liu Xin Luo Liqiang Nie Xin-Shun Xu Mohan Kankanhalli
Human-object interaction (HOI) detection aims at detecting human-object pairs
and predicting their interactions. However, the complexity of human behavior
and the diverse contexts in which these interactions occur make it challenging.
Intuitively, human-centric visual cues, such as the involved participants, the
body language, and the surrounding environment, play crucial roles in shaping
these interactions. These cues are particularly vital in interpreting unseen
interactions. In this paper, we propose three prompts with VLM to generate
human-centric visual cues within an image from multiple perspectives of humans.
To capitalize on these rich Human-Centric Visual Cues, we propose a novel
approach named HCVC for HOI detection. Particularly, we develop a
transformer-based multimodal fusion module with multitower architecture to
integrate visual cue features into the instance and interaction decoders. Our
extensive experiments and analysis validate the efficacy of leveraging the
generated human-centric visual cues for HOI detection. Notably, the
experimental results indicate the superiority of the proposed model over the
existing state-of-the-art methods on two widely used datasets.