Enhanced Distribution Alignment for Post-Training Quantization of
  Diffusion Models
Xuewen Liu Zhikai Li Junrui Xiao Qingyi Gu
Diffusion models have achieved great success in image generation tasks
through iterative noise estimation. However, the heavy denoising process and
complex neural networks hinder their low-latency applications in real-world
scenarios. Quantization can effectively reduce model complexity, and
post-training quantization (PTQ), which does not require fine-tuning, is highly
promising in accelerating the denoising process. Unfortunately, we find that
due to the highly dynamic distribution of activations in different denoising
steps, existing PTQ methods for diffusion models suffer from distribution
mismatch issues at both calibration sample level and reconstruction output
level, which makes the performance far from satisfactory, especially in low-bit
cases. In this paper, we propose Enhanced Distribution Alignment for
Post-Training Quantization of Diffusion Models (EDA-DM) to address the above
issues. Specifically, at the calibration sample level, we select calibration
samples based on the density and diversity in the latent space, thus
facilitating the alignment of their distribution with the overall samples; and
at the reconstruction output level, we propose Fine-grained Block
Reconstruction, which can align the outputs of the quantized model and the
full-precision model at different network granularity. Extensive experiments
demonstrate that EDA-DM outperforms the existing post-training quantization
frameworks in both unconditional and conditional generation scenarios. At
low-bit precision, the quantized models with our method even outperform the
full-precision models on most datasets.