Continual Learning of Diffusion Models with Generative Distillation
Sergi Masip Pau Rodriguez Tinne Tuytelaars Gido M. van de Ven
Diffusion models are powerful generative models that achieve state-of-the-art
performance in tasks such as image synthesis. However, training them demands
substantial amounts of data and computational resources. Continual learning
would allow for incrementally learning new tasks and accumulating knowledge,
thus reusing already trained models would be possible. One potentially suitable
approach is generative replay, where a copy of a generative model trained on
previous tasks produces synthetic data that are interleaved with data from the
current task. However, standard generative replay applied to diffusion models
results in a catastrophic loss in denoising capabilities. In this paper, we
propose generative distillation, an approach that distils the entire reverse
process of a diffusion model. We demonstrate that our approach significantly
improves the continual learning performance of generative replay with only a
moderate increase in the computational costs.