Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion
  Models
Wenkai Dong Song Xue Xiaoyue Duan Shumin Han
Recently large-scale language-image models (e.g., text-guided diffusion
models) have considerably improved the image generation capabilities to
generate photorealistic images in various domains. Based on this success,
current image editing methods use texts to achieve intuitive and versatile
modification of images. To edit a real image using diffusion models, one must
first invert the image to a noisy latent from which an edited image is sampled
with a target text prompt. However, most methods lack one of the following:
user-friendliness (e.g., additional masks or precise descriptions of the input
image are required), generalization to larger domains, or high fidelity to the
input image. In this paper, we design an accurate and quick inversion
technique, Prompt Tuning Inversion, for text-driven image editing.
Specifically, our proposed editing method consists of a reconstruction stage
and an editing stage. In the first stage, we encode the information of the
input image into a learnable conditional embedding via Prompt Tuning Inversion.
In the second stage, we apply classifier-free guidance to sample the edited
image, where the conditional embedding is calculated by linearly interpolating
between the target embedding and the optimized one obtained in the first stage.
This technique ensures a superior trade-off between editability and high
fidelity to the input image of our method. For example, we can change the color
of a specific object while preserving its original shape and background under
the guidance of only a target text prompt. Extensive experiments on ImageNet
demonstrate the superior editing performance of our method compared to the
state-of-the-art baselines.