Open-Vocabulary Attention Maps with Token Optimization for Semantic
  Segmentation in Diffusion Models
Pablo Marcos-Manchón Roberto Alcover-Couso Juan C. SanMiguel Jose M. Martínez
Diffusion models represent a new paradigm in text-to-image generation. Beyond
generating high-quality images from text prompts, models such as Stable
Diffusion have been successfully extended to the joint generation of semantic
segmentation pseudo-masks. However, current extensions primarily rely on
extracting attentions linked to prompt words used for image synthesis. This
approach limits the generation of segmentation masks derived from word tokens
not contained in the text prompt. In this work, we introduce Open-Vocabulary
Attention Maps (OVAM)-a training-free method for text-to-image diffusion models
that enables the generation of attention maps for any word. In addition, we
propose a lightweight optimization process based on OVAM for finding tokens
that generate accurate attention maps for an object class with a single
annotation. We evaluate these tokens within existing state-of-the-art Stable
Diffusion extensions. The best-performing model improves its mIoU from 52.1 to
86.6 for the synthetic images' pseudo-masks, demonstrating that our optimized
tokens are an efficient way to improve the performance of existing methods
without architectural changes or retraining.