OTTER: Improving Zero-Shot Classification via Optimal Transport
Changho Shin Jitian Zhao Sonia Cromp Harit Vishwakarma Frederic Sala
Popular zero-shot models suffer due to artifacts inherited from pretraining.
A particularly detrimental artifact, caused by unbalanced web-scale pretraining
data, is mismatched label distribution. Existing approaches that seek to repair
the label distribution are not suitable in zero-shot settings, as they have
incompatible requirements such as access to labeled downstream task data or
knowledge of the true label balance in the pretraining distribution. We
sidestep these challenges and introduce a simple and lightweight approach to
adjust pretrained model predictions via optimal transport. Our technique
requires only an estimate of the label distribution of a downstream task.
Theoretically, we characterize the improvement produced by our procedure under
certain mild conditions and provide bounds on the error caused by
misspecification. Empirically, we validate our method in a wide array of
zero-shot image and text classification tasks, improving accuracy by 4.8% and
15.9% on average, and beating baselines like Prior Matching -- often by
significant margins -- in 17 out of 21 datasets.