Robustness Analysis on Foundational Segmentation Models
Madeline Chantry Schiappa Sachidanand VS Yunhao Ge Ondrej Miksik Yogesh S. Rawat Vibhav Vineet
Due to the increase in computational resources and accessibility of data, an
increase in large, deep learning models trained on copious amounts of data
using self-supervised or semi-supervised learning have emerged. These
"foundation" models are often adapted to a variety of downstream tasks like
classification, object detection, and segmentation with little-to-no training
on the target dataset. In this work, we perform a robustness analysis of Visual
Foundation Models (VFMs) for segmentation tasks and compare them to supervised
models of smaller scale. We focus on robustness against real-world distribution
shift perturbations.We benchmark four state-of-the-art segmentation
architectures using 2 different datasets, COCO and ADE20K, with 17 different
perturbations with 5 severity levels each. We find interesting insights that
include (1) VFMs are not robust to compression-based corruptions, (2) while the
selected VFMs do not significantly outperform or exhibit more robustness
compared to non-VFM models, they remain competitively robust in zero-shot
evaluations, particularly when non-VFM are under supervision and (3) selected
VFMs demonstrate greater resilience to specific categories of objects, likely
due to their open-vocabulary training paradigm, a feature that non-VFM models
typically lack. We posit that the suggested robustness evaluation introduces
new requirements for foundational models, thus sparking further research to
enhance their performance.