LoRA-Enhanced Distillation on Guided Diffusion Models
Pareesa Ameneh Golnari
Diffusion models, such as Stable Diffusion (SD), offer the ability to
generate high-resolution images with diverse features, but they come at a
significant computational and memory cost. In classifier-free guided diffusion
models, prolonged inference times are attributed to the necessity of computing
two separate diffusion models at each denoising step. Recent work has shown
promise in improving inference time through distillation techniques, teaching
the model to perform similar denoising steps with reduced computations.
However, the application of distillation introduces additional memory overhead
to these already resource-intensive diffusion models, making it less practical.
  To address these challenges, our research explores a novel approach that
combines Low-Rank Adaptation (LoRA) with model distillation to efficiently
compress diffusion models. This approach not only reduces inference time but
also mitigates memory overhead, and notably decreases memory consumption even
before applying distillation. The results are remarkable, featuring a
significant reduction in inference time due to the distillation process and a
substantial 50% reduction in memory consumption. Our examination of the
generated images underscores that the incorporation of LoRA-enhanced
distillation maintains image quality and alignment with the provided prompts.
In summary, while conventional distillation tends to increase memory
consumption, LoRA-enhanced distillation offers optimization without any
trade-offs or compromises in quality.