Pathological Primitive Segmentation Based on Visual Foundation Model
  with Zero-Shot Mask Generation
Abu Bakor Hayat Arnob Xiangxue Wang Yiping Jiao Xiao Gan Wenlong Ming Jun Xu
Medical image processing usually requires a model trained with carefully
crafted datasets due to unique image characteristics and domain-specific
challenges, especially in pathology. Primitive detection and segmentation in
digitized tissue samples are essential for objective and automated diagnosis
and prognosis of cancer. SAM (Segment Anything Model) has recently been
developed to segment general objects from natural images with high accuracy,
but it requires human prompts to generate masks. In this work, we present a
novel approach that adapts pre-trained natural image encoders of SAM for
detection-based region proposals. Regions proposed by a pre-trained encoder are
sent to cascaded feature propagation layers for projection. Then, local
semantic and global context is aggregated from multi-scale for bounding box
localization and classification. Finally, the SAM decoder uses the identified
bounding boxes as essential prompts to generate a comprehensive primitive
segmentation map. The entire base framework, SAM, requires no additional
training or fine-tuning but could produce an end-to-end result for two
fundamental segmentation tasks in pathology. Our method compares with
state-of-the-art models in F1 score for nuclei detection and binary/multiclass
panoptic(bPQ/mPQ) and mask quality(dice) for segmentation quality on the
PanNuke dataset while offering end-to-end efficiency. Our model also achieves
remarkable Average Precision (+4.5%) on the secondary dataset (HuBMAP Kidney)
compared to Faster RCNN. The code is publicly available at
https://github.com/learner-codec/autoprom_sam.