Towards Realistic Unsupervised Fine-tuning with CLIP
Jian Liang Lijun Sheng Zhengbo Wang Ran He Tieniu Tan
The emergence of vision-language models (VLMs), such as CLIP, has spurred a
significant research effort towards their application for downstream supervised
learning tasks. Although some previous studies have explored the unsupervised
fine-tuning of CLIP, they often rely on prior knowledge in the form of class
names associated with ground truth labels. In this paper, we delve into a
realistic unsupervised fine-tuning scenario by assuming that the unlabeled data
might contain out-of-distribution samples from unknown classes. Furthermore, we
emphasize the importance of simultaneously enhancing out-of-distribution
detection capabilities alongside the recognition of instances associated with
predefined class labels.
  To tackle this problem, we present a simple, efficient, and effective
fine-tuning approach called Universal Entropy Optimization (UEO). UEO leverages
sample-level confidence to approximately minimize the conditional entropy of
confident instances and maximize the marginal entropy of less confident
instances. Apart from optimizing the textual prompts, UEO also incorporates
optimization of channel-wise affine transformations within the visual branch of
CLIP. Through extensive experiments conducted across 15 domains and 4 different
types of prior knowledge, we demonstrate that UEO surpasses baseline methods in
terms of both generalization and out-of-distribution detection.