Selective Guidance: Are All the Denoising Steps of Guided Diffusion
  Important?
Pareesa Ameneh Golnari Zhewei Yao Yuxiong He
This study examines the impact of optimizing the Stable Diffusion (SD) guided
inference pipeline. We propose optimizing certain denoising steps by limiting
the noise computation to conditional noise and eliminating unconditional noise
computation, thereby reducing the complexity of the target iterations by 50%.
Additionally, we demonstrate that later iterations of the SD are less sensitive
to optimization, making them ideal candidates for applying the suggested
optimization. Our experiments show that optimizing the last 20% of the
denoising loop iterations results in an 8.2% reduction in inference time with
almost no perceivable changes to the human eye. Furthermore, we found that by
extending the optimization to 50% of the last iterations, we can reduce
inference time by approximately 20.3%, while still generating visually pleasing
images.