Efficient Transfer Learning in Diffusion Models via Adversarial Noise
Xiyu Wang Baijiong Lin Daochang Liu Chang Xu
Diffusion Probabilistic Models (DPMs) have demonstrated substantial promise
in image generation tasks but heavily rely on the availability of large amounts
of training data. Previous works, like GANs, have tackled the limited data
problem by transferring pre-trained models learned with sufficient data.
However, those methods are hard to be utilized in DPMs since the distinct
differences between DPM-based and GAN-based methods, showing in the unique
iterative denoising process integral and the need for many timesteps with
no-targeted noise in DPMs. In this paper, we propose a novel DPMs-based
transfer learning method, TAN, to address the limited data problem. It includes
two strategies: similarity-guided training, which boosts transfer with a
classifier, and adversarial noise selection which adaptive chooses targeted
noise based on the input image. Extensive experiments in the context of
few-shot image generation tasks demonstrate that our method is not only
efficient but also excels in terms of image quality and diversity when compared
to existing GAN-based and DDPM-based methods.