Performance Evaluation of Segment Anything Model with Variational
  Prompting for Application to Non-Visible Spectrum Imagery
Yona Falinie A. Gaus Neelanjan Bhowmik Brian K. S. Isaac-Medina Toby P. Breckon
The Segment Anything Model (SAM) is a deep neural network foundational model
designed to perform instance segmentation which has gained significant
popularity given its zero-shot segmentation ability. SAM operates by generating
masks based on various input prompts such as text, bounding boxes, points, or
masks, introducing a novel methodology to overcome the constraints posed by
dataset-specific scarcity. While SAM is trained on an extensive dataset,
comprising ~11M images, it mostly consists of natural photographic images with
only very limited images from other modalities. Whilst the rapid progress in
visual infrared surveillance and X-ray security screening imaging technologies,
driven forward by advances in deep learning, has significantly enhanced the
ability to detect, classify and segment objects with high accuracy, it is not
evident if the SAM zero-shot capabilities can be transferred to such
modalities. This work assesses SAM capabilities in segmenting objects of
interest in the X-ray/infrared modalities. Our approach reuses the pre-trained
SAM with three different prompts: bounding box, centroid and random points. We
present quantitative/qualitative results to showcase the performance on
selected datasets. Our results show that SAM can segment objects in the X-ray
modality when given a box prompt, but its performance varies for point prompts.
Specifically, SAM performs poorly in segmenting slender objects and organic
materials, such as plastic bottles. We find that infrared objects are also
challenging to segment with point prompts given the low-contrast nature of this
modality. This study shows that while SAM demonstrates outstanding zero-shot
capabilities with box prompts, its performance ranges from moderate to poor for
point prompts, indicating that special consideration on the cross-modal
generalisation of SAM is needed when considering use on X-ray/infrared imagery.