Prompt Ensemble Self-training for Open-Vocabulary Domain Adaptation
Jiaxing Huang Jingyi Zhang Han Qiu Sheng Jin Shijian Lu
Traditional domain adaptation assumes the same vocabulary across source and
target domains, which often struggles with limited transfer flexibility and
efficiency while handling target domains with different vocabularies. Inspired
by recent vision-language models (VLMs) that enable open-vocabulary visual
recognition by reasoning on both images and texts, we study open-vocabulary
domain adaptation (OVDA), a new unsupervised domain adaptation framework that
positions a pre-trained VLM as the source model and transfers it towards
arbitrary unlabelled target domains. To this end, we design a Prompt Ensemble
Self-training (PEST) technique that exploits the synergy between vision and
language to mitigate the domain discrepancies in image and text distributions
simultaneously. Specifically, PEST makes use of the complementary property of
multiple prompts within and across vision and language modalities, which
enables joint exploitation of vision and language information and effective
learning of image-text correspondences in the unlabelled target domains.
Additionally, PEST captures temporal information via temporal prompt ensemble
which helps memorize previously learnt target information. Extensive
experiments show that PEST outperforms the state-of-the-art consistently across
10 image recognition tasks.