GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot
  Learning
Guangyue Xu Joyce Chai Parisa Kordjamshidi
Pre-trained vision-language models (VLMs) have achieved promising success in
many fields, especially with prompt learning paradigm. In this work, we propose
GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better
explore the compositional zero-shot learning (CZSL) ability of VLMs within the
prompt-based learning framework. The soft prompt in GIPCOL is structured and
consists of the prefix learnable vectors, attribute label and object label. In
addition, the attribute and object labels in the soft prompt are designated as
nodes in a compositional graph. The compositional graph is constructed based on
the compositional structure of the objects and attributes extracted from the
training data and consequently feeds the updated concept representation into
the soft prompt to capture this compositional structure for a better prompting
for CZSL. With the new prompting strategy, GIPCOL achieves state-of-the-art AUC
results on all three CZSL benchmarks, including MIT-States, UT-Zappos, and
C-GQA datasets in both closed and open settings compared to previous non-CLIP
as well as CLIP-based methods. We analyze when and why GIPCOL operates well
given the CLIP backbone and its training data limitations, and our findings
shed light on designing more effective prompts for CZSL