Contractive Diffusion Probabilistic Models
Wenpin Tang Hanyang Zhao
Diffusion probabilistic models (DPMs) have emerged as a promising technology
in generative modeling. The success of DPMs relies on two ingredients: time
reversal of Markov diffusion processes and score matching. Most existing work
implicitly assumes that score matching is close to perfect, while this
assumption is questionable. In view of possibly unguaranteed score matching, we
propose a new criterion -- the contraction of backward sampling in the design
of DPMs. This leads to a novel class of contractive DPMs (CDPMs), including
contractive Ornstein-Uhlenbeck (OU) processes and contractive sub-variance
preserving (sub-VP) stochastic differential equations (SDEs). The key insight
is that the contraction in the backward process narrows score matching errors,
as well as discretization error. Thus, the proposed CDPMs are robust to both
sources of error. Our proposal is supported by theoretical results, and is
corroborated by experiments. Notably, contractive sub-VP shows the best
performance among all known SDE-based DPMs on the CIFAR-10 dataset.