Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
Yongjian Wu Yang Zhou Jiya Saiyin Bingzheng Wei Maode Lai Jianzhong Shou Yubo Fan Yan Xu
Large-scale visual-language pre-trained models (VLPM) have proven their
excellent performance in downstream object detection for natural scenes.
However, zero-shot nuclei detection on H\&E images via VLPMs remains
underexplored. The large gap between medical images and the web-originated
text-image pairs used for pre-training makes it a challenging task. In this
paper, we attempt to explore the potential of the object-level VLPM, Grounded
Language-Image Pre-training (GLIP) model, for zero-shot nuclei detection.
Concretely, an automatic prompts design pipeline is devised based on the
association binding trait of VLPM and the image-to-text VLPM BLIP, avoiding
empirical manual prompts engineering. We further establish a self-training
framework, using the automatically designed prompts to generate the preliminary
results as pseudo labels from GLIP and refine the predicted boxes in an
iterative manner. Our method achieves a remarkable performance for label-free
nuclei detection, surpassing other comparison methods. Foremost, our work
demonstrates that the VLPM pre-trained on natural image-text pairs exhibits
astonishing potential for downstream tasks in the medical field as well. Code
will be released at https://github.com/wuyongjianCODE/VLPMNuD.