Context-Aware Prompt Tuning for Vision-Language Model with
  Dual-Alignment
Hongyu Hu Tiancheng Lin Jie Wang Zhenbang Sun Yi Xu
Large-scale vision-language models (VLMs), e.g., CLIP, learn broad visual
concepts from tedious training data, showing superb generalization ability.
Amount of prompt learning methods have been proposed to efficiently adapt the
VLMs to downstream tasks with only a few training samples. We introduce a novel
method to improve the prompt learning of vision-language models by
incorporating pre-trained large language models (LLMs), called Dual-Aligned
Prompt Tuning (DuAl-PT). Learnable prompts, like CoOp, implicitly model the
context through end-to-end training, which are difficult to control and
interpret. While explicit context descriptions generated by LLMs, like GPT-3,
can be directly used for zero-shot classification, such prompts are overly
relying on LLMs and still underexplored in few-shot domains. With DuAl-PT, we
propose to learn more context-aware prompts, benefiting from both explicit and
implicit context modeling. To achieve this, we introduce a pre-trained LLM to
generate context descriptions, and we encourage the prompts to learn from the
LLM's knowledge by alignment, as well as the alignment between prompts and
local image features. Empirically, DuAl-PT achieves superior performance on 11
downstream datasets on few-shot recognition and base-to-new generalization.
Hopefully, DuAl-PT can serve as a strong baseline. Code will be available.