Leverage Weakly Annotation to Pixel-wise Annotation via Zero-shot
  Segment Anything Model for Molecular-empowered Learning
Xueyuan Li Ruining Deng Yucheng Tang Shunxing Bao Haichun Yang Yuankai Huo
Precise identification of multiple cell classes in high-resolution Giga-pixel
whole slide imaging (WSI) is critical for various clinical scenarios. Building
an AI model for this purpose typically requires pixel-level annotations, which
are often unscalable and must be done by skilled domain experts (e.g.,
pathologists). However, these annotations can be prone to errors, especially
when distinguishing between intricate cell types (e.g., podocytes and mesangial
cells) using only visual inspection. Interestingly, a recent study showed that
lay annotators, when using extra immunofluorescence (IF) images for reference
(referred to as molecular-empowered learning), can sometimes outperform domain
experts in labeling. Despite this, the resource-intensive task of manual
delineation remains a necessity during the annotation process. In this paper,
we explore the potential of bypassing pixel-level delineation by employing the
recent segment anything model (SAM) on weak box annotation in a zero-shot
learning approach. Specifically, we harness SAM's ability to produce
pixel-level annotations from box annotations and utilize these SAM-generated
labels to train a segmentation model. Our findings show that the proposed
SAM-assisted molecular-empowered learning (SAM-L) can diminish the labeling
efforts for lay annotators by only requiring weak box annotations. This is
achieved without compromising annotation accuracy or the performance of the
deep learning-based segmentation. This research represents a significant
advancement in democratizing the annotation process for training pathological
image segmentation, relying solely on non-expert annotators.