Towards Faster Training of Diffusion Models: An Inspiration of A
  Consistency Phenomenon
Tianshuo Xu Peng Mi Ruilin Wang Yingcong Chen
Diffusion models (DMs) are a powerful generative framework that have
attracted significant attention in recent years. However, the high
computational cost of training DMs limits their practical applications. In this
paper, we start with a consistency phenomenon of DMs: we observe that DMs with
different initializations or even different architectures can produce very
similar outputs given the same noise inputs, which is rare in other generative
models. We attribute this phenomenon to two factors: (1) the learning
difficulty of DMs is lower when the noise-prediction diffusion model approaches
the upper bound of the timestep (the input becomes pure noise), where the
structural information of the output is usually generated; and (2) the loss
landscape of DMs is highly smooth, which implies that the model tends to
converge to similar local minima and exhibit similar behavior patterns. This
finding not only reveals the stability of DMs, but also inspires us to devise
two strategies to accelerate the training of DMs. First, we propose a
curriculum learning based timestep schedule, which leverages the noise rate as
an explicit indicator of the learning difficulty and gradually reduces the
training frequency of easier timesteps, thus improving the training efficiency.
Second, we propose a momentum decay strategy, which reduces the momentum
coefficient during the optimization process, as the large momentum may hinder
the convergence speed and cause oscillations due to the smoothness of the loss
landscape. We demonstrate the effectiveness of our proposed strategies on
various models and show that they can significantly reduce the training time
and improve the quality of the generated images.