Progressive Multi-modal Conditional Prompt Tuning
Xiaoyu Qiu Hao Feng Yuechen Wang Wengang Zhou Houqiang Li
Pre-trained vision-language models (VLMs) have shown remarkable
generalization capabilities via prompting, which leverages VLMs as knowledge
bases to extract information beneficial for downstream tasks. However, existing
methods primarily employ uni-modal prompting, which only engages a uni-modal
branch, failing to simultaneously adjust vision-language (V-L) features.
Additionally, the one-pass forward pipeline in VLM encoding struggles to align
V-L features that have a huge gap. Confronting these challenges, we propose a
novel method, Progressive Multi-modal conditional Prompt Tuning (ProMPT).
ProMPT exploits a recurrent structure, optimizing and aligning V-L features by
iteratively utilizing image and current encoding information. It comprises an
initialization and a multi-modal iterative evolution (MIE) module.
Initialization is responsible for encoding image and text using a VLM, followed
by a feature filter that selects text features similar to image. MIE then
facilitates multi-modal prompting through class-conditional vision prompting,
instance-conditional text prompting, and feature filtering. In each MIE
iteration, vision prompts are obtained from the filtered text features via a
vision generator, promoting image features to focus more on target object
during vision prompting. The encoded image features are fed into a text
generator to produce text prompts that are more robust to class shift. Thus,
V-L features are progressively aligned, enabling advance from coarse to exact
classifications. Extensive experiments are conducted in three settings to
evaluate the efficacy of ProMPT. The results indicate that ProMPT outperforms
existing methods on average across all settings, demonstrating its superior
generalization.