FreeA: Human-object Interaction Detection using Free Annotation Labels
Yuxiao Wang Zhenao Wei Xinyu Jiang Yu Lei Weiying Xue Jinxiu Liu Qi Liu
Recent human-object interaction (HOI) detection approaches rely on high cost
of manpower and require comprehensive annotated image datasets. In this paper,
we propose a novel self-adaption language-driven HOI detection method, termed
as FreeA, without labeling by leveraging the adaptability of CLIP to generate
latent HOI labels. To be specific, FreeA matches image features of human-object
pairs with HOI text templates, and a priori knowledge-based mask method is
developed to suppress improbable interactions. In addition, FreeA utilizes the
proposed interaction correlation matching method to enhance the likelihood of
actions related to a specified action, further refine the generated HOI labels.
Experiments on two benchmark datasets show that FreeA achieves state-of-the-art
performance among weakly supervised HOI models. Our approach is +8.58 mean
Average Precision (mAP) on HICO-DET and +1.23 mAP on V-COCO more accurate in
localizing and classifying the interactive actions than the newest weakly
model, and +1.68 mAP and +7.28 mAP than the latest weakly+ model, respectively.
Code will be available at https://drliuqi.github.io/.