Improving Adaptability and Generalizability of Efficient Transfer
  Learning for Vision-Language Models
Yongjin Yang Jongwoo Ko Se-Young Yun
Vision-Language Models (VLMs) like CLIP have demonstrated remarkable
applicability across a variety of downstream tasks, including zero-shot image
classification. Recently, the use of prompts or adapters for efficient transfer
learning has gained significant attention for effectively adapting to
downstream tasks. However, the roles of vision and text prompts, as well as
adapters in terms of generalization and transfer difficulty, have been
overlooked, limiting performance on unseen tasks. In this paper, we empirically
analyze how VLMs behave when using vision and text prompts, adapters, and a
combination of these components, marking a novel exploration by our study. Our
observations find that utilizing vision prompts for class separability and text
adapters for task adaptation is crucial for adaptability and generalizability.
Moreover, to improve generalization across every domain, we propose an adaptive
ensemble method that effectively combines the general knowledge of VLMs with
task-specific knowledge according to transfer difficulty. Upon experimenting
with extensive benchmarks, our method consistently outperforms all baselines,
particularly on unseen tasks, demonstrating the effectiveness of our proposed
approach.