An Item is Worth a Prompt: Versatile Image Editing with Disentangled
  Control
Aosong Feng Weikang Qiu Jinbin Bai Kaicheng Zhou Zhen Dong Xiao Zhang Rex Ying Leandros Tassiulas
Building on the success of text-to-image diffusion models (DPMs), image
editing is an important application to enable human interaction with
AI-generated content. Among various editing methods, editing within the prompt
space gains more attention due to its capacity and simplicity of controlling
semantics. However, since diffusion models are commonly pretrained on
descriptive text captions, direct editing of words in text prompts usually
leads to completely different generated images, violating the requirements for
image editing. On the other hand, existing editing methods usually consider
introducing spatial masks to preserve the identity of unedited regions, which
are usually ignored by DPMs and therefore lead to inharmonic editing results.
Targeting these two challenges, in this work, we propose to disentangle the
comprehensive image-prompt interaction into several item-prompt interactions,
with each item linked to a special learned prompt. The resulting framework,
named D-Edit, is based on pretrained diffusion models with cross-attention
layers disentangled and adopts a two-step optimization to build item-prompt
associations. Versatile image editing can then be applied to specific items by
manipulating the corresponding prompts. We demonstrate state-of-the-art results
in four types of editing operations including image-based, text-based,
mask-based editing, and item removal, covering most types of editing
applications, all within a single unified framework. Notably, D-Edit is the
first framework that can (1) achieve item editing through mask editing and (2)
combine image and text-based editing. We demonstrate the quality and
versatility of the editing results for a diverse collection of images through
both qualitative and quantitative evaluations.