UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images
Zhen Chen Qing Xu Xinyu Liu Yixuan Yuan
In digital pathology, precise nuclei segmentation is pivotal yet challenged
by the diversity of tissue types, staining protocols, and imaging conditions.
Recently, the segment anything model (SAM) revealed overwhelming performance in
natural scenarios and impressive adaptation to medical imaging. Despite these
advantages, the reliance of labor-intensive manual annotation as segmentation
prompts severely hinders their clinical applicability, especially for nuclei
image analysis containing massive cells where dense manual prompts are
impractical. To overcome the limitations of current SAM methods while retaining
the advantages, we propose the Universal prompt-free SAM framework for Nuclei
segmentation (UN-SAM), by providing a fully automated solution with remarkable
generalization capabilities. Specifically, to eliminate the labor-intensive
requirement of per-nuclei annotations for prompt, we devise a multi-scale
Self-Prompt Generation (SPGen) module to revolutionize clinical workflow by
automatically generating high-quality mask hints to guide the segmentation
tasks. Moreover, to unleash the generalization capability of SAM across a
variety of nuclei images, we devise a Domain-adaptive Tuning Encoder
(DT-Encoder) to seamlessly harmonize visual features with domain-common and
domain-specific knowledge, and further devise a Domain Query-enhanced Decoder
(DQ-Decoder) by leveraging learnable domain queries for segmentation decoding
in different nuclei domains. Extensive experiments prove that UN-SAM with
exceptional performance surpasses state-of-the-arts in nuclei instance and
semantic segmentation, especially the generalization capability in zero-shot
scenarios. The source code is available at
https://github.com/CUHK-AIM-Group/UN-SAM.