Compositional Kronecker Context Optimization for Vision-Language Models
Kun Ding Xiaohui Li Qiang Yu Ying Wang Haojian Zhang Shiming Xiang
Context Optimization (CoOp) has emerged as a simple yet effective technique
for adapting CLIP-like vision-language models to downstream image recognition
tasks. Nevertheless, learning compact context with satisfactory base-to-new,
domain and cross-task generalization ability while adapting to new tasks is
still a challenge. To tackle such a challenge, we propose a lightweight yet
generalizable approach termed Compositional Kronecker Context Optimization
(CK-CoOp). Technically, the prompt's context words in CK-CoOp are learnable
vectors, which are crafted by linearly combining base vectors sourced from a
dictionary. These base vectors consist of a non-learnable component obtained by
quantizing the weights in the token embedding layer, and a learnable component
constructed by applying Kronecker product on several learnable tiny matrices.
Intuitively, the compositional structure mitigates the risk of overfitting on
training data by remembering more pre-trained knowledge. Meantime, the
Kronecker product breaks the non-learnable restrictions of the dictionary,
thereby enhancing representation ability with minimal additional parameters.
Extensive experiments confirm that CK-CoOp achieves state-of-the-art
performance under base-to-new, domain and cross-task generalization evaluation,
but also has the metrics of fewer learnable parameters and efficient training
and inference speed.