Training-Free Open-Vocabulary Segmentation with Offline
  Diffusion-Augmented Prototype Generation
Luca Barsellotti Roberto Amoroso Marcella Cornia Lorenzo Baraldi Rita Cucchiara
Open-vocabulary semantic segmentation aims at segmenting arbitrary categories
expressed in textual form. Previous works have trained over large amounts of
image-caption pairs to enforce pixel-level multimodal alignments. However,
captions provide global information about the semantics of a given image but
lack direct localization of individual concepts. Further, training on
large-scale datasets inevitably brings significant computational costs. In this
paper, we propose FreeDA, a training-free diffusion-augmented method for
open-vocabulary semantic segmentation, which leverages the ability of diffusion
models to visually localize generated concepts and local-global similarities to
match class-agnostic regions with semantic classes. Our approach involves an
offline stage in which textual-visual reference embeddings are collected,
starting from a large set of captions and leveraging visual and semantic
contexts. At test time, these are queried to support the visual matching
process, which is carried out by jointly considering class-agnostic regions and
global semantic similarities. Extensive analyses demonstrate that FreeDA
achieves state-of-the-art performance on five datasets, surpassing previous
methods by more than 7.0 average points in terms of mIoU and without requiring
any training.