Open-Set Domain Adaptation with Visual-Language Foundation Models
Qing Yu Go Irie Kiyoharu Aizawa
Unsupervised domain adaptation (UDA) has proven to be very effective in
transferring knowledge obtained from a source domain with labeled data to a
target domain with unlabeled data. Owing to the lack of labeled data in the
target domain and the possible presence of unknown classes, open-set domain
adaptation (ODA) has emerged as a potential solution to identify these classes
during the training phase. Although existing ODA approaches aim to solve the
distribution shifts between the source and target domains, most methods
fine-tuned ImageNet pre-trained models on the source domain with the adaptation
on the target domain. Recent visual-language foundation models (VLFM), such as
Contrastive Language-Image Pre-Training (CLIP), are robust to many distribution
shifts and, therefore, should substantially improve the performance of ODA. In
this work, we explore generic ways to adopt CLIP, a popular VLFM, for ODA. We
investigate the performance of zero-shot prediction using CLIP, and then
propose an entropy optimization strategy to assist the ODA models with the
outputs of CLIP. The proposed approach achieves state-of-the-art results on
various benchmarks, demonstrating its effectiveness in addressing the ODA
problem.