Layered Rendering Diffusion Model for Zero-Shot Guided Image Synthesis
Zipeng Qi Guoxi Huang Zebin Huang Qin Guo Jinwen Chen Junyu Han Jian Wang Gang Zhang Lufei Liu Errui Ding Jingdong Wang
This paper introduces innovative solutions to enhance spatial controllability
in diffusion models reliant on text queries. We present two key innovations:
Vision Guidance and the Layered Rendering Diffusion (LRDiff) framework. Vision
Guidance, a spatial layout condition, acts as a clue in the perturbed
distribution, greatly narrowing down the search space, to focus on the image
sampling process adhering to the spatial layout condition. The LRDiff framework
constructs an image-rendering process with multiple layers, each of which
applies the vision guidance to instructively estimate the denoising direction
for a single object. Such a layered rendering strategy effectively prevents
issues like unintended conceptual blending or mismatches, while allowing for
more coherent and contextually accurate image synthesis. The proposed method
provides a more efficient and accurate means of synthesising images that align
with specific spatial and contextual requirements. We demonstrate through our
experiments that our method provides better results than existing techniques
both quantitatively and qualitatively. We apply our method to three practical
applications: bounding box-to-image, semantic mask-to-image and image editing.