Annotation Free Semantic Segmentation with Vision Foundation Models
Soroush Seifi Daniel Olmeda Reino Fabien Despinoy Rahaf Aljundi
Semantic Segmentation is one of the most challenging vision tasks, usually
requiring large amounts of training data with expensive pixel-level
annotations. With the success of foundation models and especially
vision-language models, recent works attempt to achieve zero-shot semantic
segmentation while requiring either large scale training or additional
image/pixel-level annotations. In this work, we build a lightweight module on
top of a self-supervised pretrained vision encoder to align patch features with
a pre-trained text encoder. Importantly, we generate free annotations for any
semantic segmentation dataset using existing foundation models and train our
alignment module cost free. We use CLIP to detect objects and SAM to generate
high quality object masks. Our approach can bring language-based semantics to
any pre-trained vision encoder with minimal training. Our module is
lightweight, uses foundation models as a sole source of supervision and shows
impressive generalization capability from little training data with no
annotation.