Eliminating Lipschitz Singularities in Diffusion Models
Zhantao Yang Ruili Feng Han Zhang Yujun Shen Kai Zhu Lianghua Huang Yifei Zhang Yu Liu Deli Zhao Jingren Zhou Fan Cheng
Diffusion models, which employ stochastic differential equations to sample
images through integrals, have emerged as a dominant class of generative
models. However, the rationality of the diffusion process itself receives
limited attention, leaving the question of whether the problem is well-posed
and well-conditioned. In this paper, we uncover a vexing propensity of
diffusion models: they frequently exhibit the infinite Lipschitz near the zero
point of timesteps. This poses a threat to the stability and accuracy of the
diffusion process, which relies on integral operations. We provide a
comprehensive evaluation of the issue from both theoretical and empirical
perspectives. To address this challenge, we propose a novel approach, dubbed
E-TSDM, which eliminates the Lipschitz singularity of the diffusion model near
zero. Remarkably, our technique yields a substantial improvement in
performance, e.g., on the high-resolution FFHQ dataset ($256\times256$).
Moreover, as a byproduct of our method, we manage to achieve a dramatic
reduction in the Frechet Inception Distance of other acceleration methods
relying on network Lipschitz, including DDIM and DPM-Solver, by over 33$\%$. We
conduct extensive experiments on diverse datasets to validate our theory and
method. Our work not only advances the understanding of the general diffusion
process, but also provides insights for the design of diffusion models.