Not All Steps are Created Equal: Selective Diffusion Distillation for
  Image Manipulation
Luozhou Wang Shuai Yang Shu Liu Ying-cong Chen
Conditional diffusion models have demonstrated impressive performance in
image manipulation tasks. The general pipeline involves adding noise to the
image and then denoising it. However, this method faces a trade-off problem:
adding too much noise affects the fidelity of the image while adding too little
affects its editability. This largely limits their practical applicability. In
this paper, we propose a novel framework, Selective Diffusion Distillation
(SDD), that ensures both the fidelity and editability of images. Instead of
directly editing images with a diffusion model, we train a feedforward image
manipulation network under the guidance of the diffusion model. Besides, we
propose an effective indicator to select the semantic-related timestep to
obtain the correct semantic guidance from the diffusion model. This approach
successfully avoids the dilemma caused by the diffusion process. Our extensive
experiments demonstrate the advantages of our framework. Code is released at
https://github.com/AndysonYs/Selective-Diffusion-Distillation.