Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge
  Distillation at Multiple Levels
Bo Wan Tinne Tuytelaars
In this paper, we investigate the task of zero-shot human-object interaction
(HOI) detection, a novel paradigm for identifying HOIs without the need for
task-specific annotations. To address this challenging task, we employ CLIP, a
large-scale pre-trained vision-language model (VLM), for knowledge distillation
on multiple levels. Specifically, we design a multi-branch neural network that
leverages CLIP for learning HOI representations at various levels, including
global images, local union regions encompassing human-object pairs, and
individual instances of humans or objects. To train our model, CLIP is utilized
to generate HOI scores for both global images and local union regions that
serve as supervision signals. The extensive experiments demonstrate the
effectiveness of our novel multi-level CLIP knowledge integration strategy.
Notably, the model achieves strong performance, which is even comparable with
some fully-supervised and weakly-supervised methods on the public HICO-DET
benchmark.