FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion
  Models
Barbara Toniella Corradini Mustafa Shukor Paul Couairon Guillaume Couairon Franco Scarselli Matthieu Cord
Foundation models have exhibited unprecedented capabilities in tackling many
domains and tasks. Models such as CLIP are currently widely used to bridge
cross-modal representations, and text-to-image diffusion models are arguably
the leading models in terms of realistic image generation. Image generative
models are trained on massive datasets that provide them with powerful internal
spatial representations. In this work, we explore the potential benefits of
such representations, beyond image generation, in particular, for dense visual
prediction tasks. We focus on the task of image segmentation, which is
traditionally solved by training models on closed-vocabulary datasets, with
pixel-level annotations. To avoid the annotation cost or training large
diffusion models, we constraint our setup to be zero-shot and training-free. In
a nutshell, our pipeline leverages different and relatively small-sized,
open-source foundation models for zero-shot open-vocabulary segmentation. The
pipeline is as follows: the image is passed to both a captioner model (i.e.
BLIP) and a diffusion model (i.e., Stable Diffusion Model) to generate a text
description and visual representation, respectively. The features are clustered
and binarized to obtain class agnostic masks for each object. These masks are
then mapped to a textual class, using the CLIP model to support
open-vocabulary. Finally, we add a refinement step that allows to obtain a more
precise segmentation mask. Our approach (dubbed FreeSeg-Diff), which does not
rely on any training, outperforms many training-based approaches on both Pascal
VOC and COCO datasets. In addition, we show very competitive results compared
to the recent weakly-supervised segmentation approaches. We provide
comprehensive experiments showing the superiority of diffusion model features
compared to other pretrained models. Project page:
https://bcorrad.github.io/freesegdiff/