Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning
Shuo Yang Zirui Shang Yongqi Wang Derong Deng Hongwei Chen Qiyuan Cheng Xinxiao Wu
This paper proposes a novel framework for multi-label image recognition
without any training data, called data-free framework, which uses knowledge of
pre-trained Large Language Model (LLM) to learn prompts to adapt pretrained
Vision-Language Model (VLM) like CLIP to multilabel classification. Through
asking LLM by well-designed questions, we acquire comprehensive knowledge about
characteristics and contexts of objects, which provides valuable text
descriptions for learning prompts. Then we propose a hierarchical prompt
learning method by taking the multi-label dependency into consideration,
wherein a subset of category-specific prompt tokens are shared when the
corresponding objects exhibit similar attributes or are more likely to
co-occur. Benefiting from the remarkable alignment between visual and
linguistic semantics of CLIP, the hierarchical prompts learned from text
descriptions are applied to perform classification of images during inference.
Our framework presents a new way to explore the synergies between multiple
pre-trained models for novel category recognition. Extensive experiments on
three public datasets (MS-COCO, VOC2007, and NUS-WIDE) demonstrate that our
method achieves better results than the state-of-the-art methods, especially
outperforming the zero-shot multi-label recognition methods by 4.7% in mAP on
MS-COCO.