InstructEdit: Improving Automatic Masks for Diffusion-based Image
  Editing With User Instructions
Qian Wang Biao Zhang Michael Birsak Peter Wonka
Recent works have explored text-guided image editing using diffusion models
and generated edited images based on text prompts. However, the models struggle
to accurately locate the regions to be edited and faithfully perform precise
edits. In this work, we propose a framework termed InstructEdit that can do
fine-grained editing based on user instructions. Our proposed framework has
three components: language processor, segmenter, and image editor. The first
component, the language processor, processes the user instruction using a large
language model. The goal of this processing is to parse the user instruction
and output prompts for the segmenter and captions for the image editor. We
adopt ChatGPT and optionally BLIP2 for this step. The second component, the
segmenter, uses the segmentation prompt provided by the language processor. We
employ a state-of-the-art segmentation framework Grounded Segment Anything to
automatically generate a high-quality mask based on the segmentation prompt.
The third component, the image editor, uses the captions from the language
processor and the masks from the segmenter to compute the edited image. We
adopt Stable Diffusion and the mask-guided generation from DiffEdit for this
purpose. Experiments show that our method outperforms previous editing methods
in fine-grained editing applications where the input image contains a complex
object or multiple objects. We improve the mask quality over DiffEdit and thus
improve the quality of edited images. We also show that our framework can
accept multiple forms of user instructions as input. We provide the code at
https://github.com/QianWangX/InstructEdit.