The devil is in the object boundary: towards annotation-free instance
  segmentation using Foundation Models
Cheng Shi Sibei Yang
Foundation models, pre-trained on a large amount of data have demonstrated
impressive zero-shot capabilities in various downstream tasks. However, in
object detection and instance segmentation, two fundamental computer vision
tasks heavily reliant on extensive human annotations, foundation models such as
SAM and DINO struggle to achieve satisfactory performance. In this study, we
reveal that the devil is in the object boundary, \textit{i.e.}, these
foundation models fail to discern boundaries between individual objects. For
the first time, we probe that CLIP, which has never accessed any instance-level
annotations, can provide a highly beneficial and strong instance-level boundary
prior in the clustering results of its particular intermediate layer. Following
this surprising observation, we propose $\textbf{Zip}$ which $\textbf{Z}$ips up
CL$\textbf{ip}$ and SAM in a novel classification-first-then-discovery
pipeline, enabling annotation-free, complex-scene-capable, open-vocabulary
object detection and instance segmentation. Our Zip significantly boosts SAM's
mask AP on COCO dataset by 12.5% and establishes state-of-the-art performance
in various settings, including training-free, self-training, and
label-efficient finetuning. Furthermore, annotation-free Zip even achieves
comparable performance to the best-performing open-vocabulary object detecters
using base annotations. Code is released at
https://github.com/ChengShiest/Zip-Your-CLIP