Exploring Simple Open-Vocabulary Semantic Segmentation
Zihang Lai
Open-vocabulary semantic segmentation models aim to accurately assign a
semantic label to each pixel in an image from a set of arbitrary
open-vocabulary texts. In order to learn such pixel-level alignment, current
approaches typically rely on a combination of (i) image-level VL model (e.g.
CLIP), (ii) ground truth masks, and (iii) custom grouping encoders. In this
paper, we introduce S-Seg, a novel model that can achieve surprisingly strong
performance without depending on any of the above elements. S-Seg leverages
pseudo-mask and language to train a MaskFormer, and can be easily trained from
publicly available image-text datasets. Contrary to prior works, our model
directly trains for pixel-level features and language alignment. Once trained,
S-Seg generalizes well to multiple testing datasets without requiring
fine-tuning. In addition, S-Seg has the extra benefits of scalability with data
and consistently improvement when augmented with self-training. We believe that
our simple yet effective approach will serve as a solid baseline for future
research.