Denoising Task Difficulty-based Curriculum for Training Diffusion Models
Jin-Young Kim Hyojun Go Soonwoo Kwon Hyun-Gyoon Kim
Diffusion-based generative models have emerged as powerful tools in the realm
of generative modeling. Despite extensive research on denoising across various
timesteps and noise levels, a conflict persists regarding the relative
difficulties of the denoising tasks. While various studies argue that lower
timesteps present more challenging tasks, others contend that higher timesteps
are more difficult. To address this conflict, our study undertakes a
comprehensive examination of task difficulties, focusing on convergence
behavior and changes in relative entropy between consecutive probability
distributions across timesteps. Our observational study reveals that denoising
at earlier timesteps poses challenges characterized by slower convergence and
higher relative entropy, indicating increased task difficulty at these lower
timesteps. Building on these observations, we introduce an easy-to-hard
learning scheme, drawing from curriculum learning, to enhance the training
process of diffusion models. By organizing timesteps or noise levels into
clusters and training models with descending orders of difficulty, we
facilitate an order-aware training regime, progressing from easier to harder
denoising tasks, thereby deviating from the conventional approach of training
diffusion models simultaneously across all timesteps. Our approach leads to
improved performance and faster convergence by leveraging the benefits of
curriculum learning, while maintaining orthogonality with existing improvements
in diffusion training techniques. We validate these advantages through
comprehensive experiments in image generation tasks, including unconditional,
class-conditional, and text-to-image generation.