Zero-shot spatial layout conditioning for text-to-image diffusion models
Guillaume Couairon Marlène Careil Matthieu Cord Stéphane Lathuilière Jakob Verbeek
Large-scale text-to-image diffusion models have significantly improved the
state of the art in generative image modelling and allow for an intuitive and
powerful user interface to drive the image generation process. Expressing
spatial constraints, e.g. to position specific objects in particular locations,
is cumbersome using text; and current text-based image generation models are
not able to accurately follow such instructions. In this paper we consider
image generation from text associated with segments on the image canvas, which
combines an intuitive natural language interface with precise spatial control
over the generated content. We propose ZestGuide, a zero-shot segmentation
guidance approach that can be plugged into pre-trained text-to-image diffusion
models, and does not require any additional training. It leverages implicit
segmentation maps that can be extracted from cross-attention layers, and uses
them to align the generation with input masks. Our experimental results combine
high image quality with accurate alignment of generated content with input
segmentations, and improve over prior work both quantitatively and
qualitatively, including methods that require training on images with
corresponding segmentations. Compared to Paint with Words, the previous
state-of-the art in image generation with zero-shot segmentation conditioning,
we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.