Weak Distribution Detectors Lead to Stronger Generalizability of
  Vision-Language Prompt Tuning
Kun Ding Haojian Zhang Qiang Yu Ying Wang Shiming Xiang Chunhong Pan
We propose a generalized method for boosting the generalization ability of
pre-trained vision-language models (VLMs) while fine-tuning on downstream
few-shot tasks. The idea is realized by exploiting out-of-distribution (OOD)
detection to predict whether a sample belongs to a base distribution or a novel
distribution and then using the score generated by a dedicated competition
based scoring function to fuse the zero-shot and few-shot classifier. The fused
classifier is dynamic, which will bias towards the zero-shot classifier if a
sample is more likely from the distribution pre-trained on, leading to improved
base-to-novel generalization ability. Our method is performed only in test
stage, which is applicable to boost existing methods without time-consuming
re-training. Extensive experiments show that even weak distribution detectors
can still improve VLMs' generalization ability. Specifically, with the help of
OOD detectors, the harmonic mean of CoOp and ProGrad increase by 2.6 and 1.5
percentage points over 11 recognition datasets in the base-to-novel setting.