VLM-CPL: Consensus Pseudo Labels from Vision-Language Models for Human
  Annotation-Free Pathological Image Classification
Lanfeng Zhong Xin Liao Shaoting Zhang Xiaofan Zhang Guotai Wang
Despite that deep learning methods have achieved remarkable performance in
pathology image classification, they heavily rely on labeled data, demanding
extensive human annotation efforts. In this study, we present a novel human
annotation-free method for pathology image classification by leveraging
pre-trained Vision-Language Models (VLMs). Without human annotation, pseudo
labels of the training set are obtained by utilizing the zero-shot inference
capabilities of VLM, which may contain a lot of noise due to the domain shift
between the pre-training data and the target dataset. To address this issue, we
introduce VLM-CPL, a novel approach based on consensus pseudo labels that
integrates two noisy label filtering techniques with a semi-supervised learning
strategy. Specifically, we first obtain prompt-based pseudo labels with
uncertainty estimation by zero-shot inference with the VLM using multiple
augmented views of an input. Then, by leveraging the feature representation
ability of VLM, we obtain feature-based pseudo labels via sample clustering in
the feature space. Prompt-feature consensus is introduced to select reliable
samples based on the consensus between the two types of pseudo labels. By
rejecting low-quality pseudo labels, we further propose High-confidence Cross
Supervision (HCS) to learn from samples with reliable pseudo labels and the
remaining unlabeled samples. Experimental results showed that our method
obtained an accuracy of 87.1% and 95.1% on the HPH and LC25K datasets,
respectively, and it largely outperformed existing zero-shot classification and
noisy label learning methods. The code is available at
https://github.com/lanfz2000/VLM-CPL.