Detector Guidance for Multi-Object Text-to-Image Generation
Luping Liu Zijian Zhang Yi Ren Rongjie Huang Xiang Yin Zhou Zhao
Diffusion models have demonstrated impressive performance in text-to-image
generation. They utilize a text encoder and cross-attention blocks to infuse
textual information into images at a pixel level. However, their capability to
generate images with text containing multiple objects is still restricted.
Previous works identify the problem of information mixing in the CLIP text
encoder and introduce the T5 text encoder or incorporate strong prior knowledge
to assist with the alignment. We find that mixing problems also occur on the
image side and in the cross-attention blocks. The noisy images can cause
different objects to appear similar, and the cross-attention blocks inject
information at a pixel level, leading to leakage of global object understanding
and resulting in object mixing. In this paper, we introduce Detector Guidance
(DG), which integrates a latent object detection model to separate different
objects during the generation process. DG first performs latent object
detection on cross-attention maps (CAMs) to obtain object information. Based on
this information, DG then masks conflicting prompts and enhances related
prompts by manipulating the following CAMs. We evaluate the effectiveness of DG
using Stable Diffusion on COCO, CC, and a novel multi-related object benchmark,
MRO. Human evaluations demonstrate that DG provides an 8-22\% advantage in
preventing the amalgamation of conflicting concepts and ensuring that each
object possesses its unique region without any human involvement and additional
iterations. Our implementation is available at
\url{https://github.com/luping-liu/Detector-Guidance}.