Prompt Tuning for Zero-shot Compositional Learning
Lingyu Zhang Ting Hua Yilin Shen Hongxia Jin
Open World Compositional Zero-Shot Learning (OW-CZSL) is known to be an
extremely challenging task, which aims to recognize unseen compositions formed
from seen attributes and objects without any prior assumption of the output
space. In order to achieve this goal, a model has to be "smart" and
"knowledgeable". To be smart, a model should be good at reasoning the
interactions between attributes and objects from the seen compositions. While
"knowledgeable" means the model owns "common sense" to the open world that can
"foresee" some features of the unseen compositions. Most previous work focuses
on the "smart" part, while few of them provided an effective solution to
achieve the "knowledgeable" goal. In this paper, we proposed a framework named
Multi-Modal Prompt Tuning (MMPT) to inherit the "knowledgeable" property from
the large pre-trained vision-language model. Extensive experiments show that
our proposed MMPT obtains new state-of-the-art results in OW-CZSL task. On the
UT-Zappos dataset, MMPT pushes the AUC score to $29.8$, while the previous best
score is $26.5$. On the more challenging MIT-States dataset, the AUC score of
MMPT is 1.5 times better than the current state-of-the-art.