SOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Shengcao Cao Jiuxiang Gu Jason Kuen Hao Tan Ruiyi Zhang Handong Zhao Ani Nenkova Liang-Yan Gui Tong Sun Yu-Xiong Wang
Open-world entity segmentation, as an emerging computer vision task, aims at
segmenting entities in images without being restricted by pre-defined classes,
offering impressive generalization capabilities on unseen images and concepts.
Despite its promise, existing entity segmentation methods like Segment Anything
Model (SAM) rely heavily on costly expert annotators. This work presents
Self-supervised Open-world Hierarchical Entity Segmentation (SOHES), a novel
approach that eliminates the need for human annotations. SOHES operates in
three phases: self-exploration, self-instruction, and self-correction. Given a
pre-trained self-supervised representation, we produce abundant high-quality
pseudo-labels through visual feature clustering. Then, we train a segmentation
model on the pseudo-labels, and rectify the noises in pseudo-labels via a
teacher-student mutual-learning procedure. Beyond segmenting entities, SOHES
also captures their constituent parts, providing a hierarchical understanding
of visual entities. Using raw images as the sole training data, our method
achieves unprecedented performance in self-supervised open-world segmentation,
marking a significant milestone towards high-quality open-world entity
segmentation in the absence of human-annotated masks. Project page:
https://SOHES.github.io.