Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt
  Learning with Data-Dependent Prior
Youngjae Cho HeeSun Bae Seungjae Shin Yeo Dong Youn Weonyoung Joo Il-Chul Moon
Recent Vision-Language Pretrained (VLP) models have become the backbone for
many downstream tasks, but they are utilized as frozen model without learning.
Prompt learning is a method to improve the pre-trained VLP model by adding a
learnable context vector to the inputs of the text encoder. In a few-shot
learning scenario of the downstream task, MLE training can lead the context
vector to over-fit dominant image features in the training data. This
overfitting can potentially harm the generalization ability, especially in the
presence of a distribution shift between the training and test dataset. This
paper presents a Bayesian-based framework of prompt learning, which could
alleviate the overfitting issues on few-shot learning application and increase
the adaptability of prompts on unseen instances. Specifically, modeling
data-dependent prior enhances the adaptability of text features for both seen
and unseen image features without the trade-off of performance between them.
Based on the Bayesian framework, we utilize the Wasserstein Gradient Flow in
the estimation of our target posterior distribution, which enables our prompt
to be flexible in capturing the complex modes of image features. We demonstrate
the effectiveness of our method on benchmark datasets for several experiments
by showing statistically significant improvements on performance compared to
existing methods. The code is available at https://github.com/youngjae-cho/APP.