Temporally-Extended Prompts Optimization for SAM in Interactive Medical
  Image Segmentation
Chuyun Shen Wenhao Li Ya Zhang Xiangfeng Wang
The Segmentation Anything Model (SAM) has recently emerged as a foundation
model for addressing image segmentation. Owing to the intrinsic complexity of
medical images and the high annotation cost, the medical image segmentation
(MIS) community has been encouraged to investigate SAM's zero-shot capabilities
to facilitate automatic annotation. Inspired by the extraordinary
accomplishments of interactive medical image segmentation (IMIS) paradigm, this
paper focuses on assessing the potential of SAM's zero-shot capabilities within
the IMIS paradigm to amplify its benefits in the MIS domain. Regrettably, we
observe that SAM's vulnerability to prompt forms (e.g., points, bounding boxes)
becomes notably pronounced in IMIS. This leads us to develop a framework that
adaptively offers suitable prompt forms for human experts. We refer to the
framework above as temporally-extended prompts optimization (TEPO) and model it
as a Markov decision process, solvable through reinforcement learning.
Numerical experiments on the standardized benchmark BraTS2020 demonstrate that
the learned TEPO agent can further enhance SAM's zero-shot capability in the
MIS context.