DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified &
  Accurate Image Editing
Yueru Jia Yuhui Yuan Aosong Cheng Chuke Wang Ji Li Huizhu Jia Shanghang Zhang
Recently, how to achieve precise image editing has attracted increasing
attention, especially given the remarkable success of text-to-image generation
models. To unify various spatial-aware image editing abilities into one
framework, we adopt the concept of layers from the design domain to manipulate
objects flexibly with various operations. The key insight is to transform the
spatial-aware image editing task into a combination of two sub-tasks:
multi-layered latent decomposition and multi-layered latent fusion. First, we
segment the latent representations of the source images into multiple layers,
which include several object layers and one incomplete background layer that
necessitates reliable inpainting. To avoid extra tuning, we further explore the
inner inpainting ability within the self-attention mechanism. We introduce a
key-masking self-attention scheme that can propagate the surrounding context
information into the masked region while mitigating its impact on the regions
outside the mask. Second, we propose an instruction-guided latent fusion that
pastes the multi-layered latent representations onto a canvas latent. We also
introduce an artifact suppression scheme in the latent space to enhance the
inpainting quality. Due to the inherent modular advantages of such
multi-layered representations, we can achieve accurate image editing, and we
demonstrate that our approach consistently surpasses the latest spatial editing
methods, including Self-Guidance and DiffEditor. Last, we show that our
approach is a unified framework that supports various accurate image editing
tasks on more than six different editing tasks.