Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language
  Fusion
Xiao Wang Jiandong Jin Chenglong Li Jin Tang Cheng Zhang Wei Wang
Existing pedestrian attribute recognition (PAR) algorithms adopt pre-trained
CNN (e.g., ResNet) as their backbone network for visual feature learning, which
might obtain sub-optimal results due to the insufficient employment of the
relations between pedestrian images and attribute labels. In this paper, we
formulate PAR as a vision-language fusion problem and fully exploit the
relations between pedestrian images and attribute labels. Specifically, the
attribute phrases are first expanded into sentences, and then the pre-trained
vision-language model CLIP is adopted as our backbone for feature embedding of
visual images and attribute descriptions. The contrastive learning objective
connects the vision and language modalities well in the CLIP-based feature
space, and the Transformer layers used in CLIP can capture the long-range
relations between pixels. Then, a multi-modal Transformer is adopted to fuse
the dual features effectively and feed-forward network is used to predict
attributes. To optimize our network efficiently, we propose the region-aware
prompt tuning technique to adjust very few parameters (i.e., only the prompt
vectors and classification heads) and fix both the pre-trained VL model and
multi-modal Transformer. Our proposed PAR algorithm only adjusts 0.75%
learnable parameters compared with the fine-tuning strategy. It also achieves
new state-of-the-art performance on both standard and zero-shot settings for
PAR, including RAPv1, RAPv2, WIDER, PA100K, and PETA-ZS, RAP-ZS datasets. The
source code and pre-trained models will be released on
https://github.com/Event-AHU/OpenPAR.