Machine Vision Therapy: Multimodal Large Language Models Can Enhance
  Visual Robustness via Denoising In-Context Learning
Zhuo Huang Chang Liu Yinpeng Dong Hang Su Shibao Zheng Tongliang Liu
Although vision models such as Contrastive Language-Image Pre-Training (CLIP)
show impressive generalization performance, their zero-shot robustness is still
limited under Out-of-Distribution (OOD) scenarios without fine-tuning. Instead
of undesirably providing human supervision as commonly done, it is possible to
take advantage of Multi-modal Large Language Models (MLLMs) that hold powerful
visual understanding abilities. However, MLLMs are shown to struggle with
vision problems due to the incompatibility of tasks, thus hindering their
utilization. In this paper, we propose to effectively leverage MLLMs to conduct
Machine Vision Therapy which aims to rectify the noisy predictions from vision
models. By fine-tuning with the denoised labels, the learning model performance
can be boosted in an unsupervised manner. To solve the incompatibility issue,
we propose a novel Denoising In-Context Learning (DICL) strategy to align
vision tasks with MLLMs. Concretely, by estimating a transition matrix that
captures the probability of one class being confused with another, an
instruction containing a correct exemplar and an erroneous one from the most
probable noisy class can be constructed. Such an instruction can help any MLLMs
with ICL ability to detect and rectify incorrect predictions of vision models.
Through extensive experiments on ImageNet, WILDS, DomainBed, and other OOD
datasets, we carefully validate the quantitative and qualitative effectiveness
of our method. Our code is available at
https://github.com/tmllab/Machine_Vision_Therapy.