Towards Memorization-Free Diffusion Models
Chen Chen Daochang Liu Chang Xu
Pretrained diffusion models and their outputs are widely accessible due to
their exceptional capacity for synthesizing high-quality images and their
open-source nature. The users, however, may face litigation risks owing to the
models' tendency to memorize and regurgitate training data during inference. To
address this, we introduce Anti-Memorization Guidance (AMG), a novel framework
employing three targeted guidance strategies for the main causes of
memorization: image and caption duplication, and highly specific user prompts.
Consequently, AMG ensures memorization-free outputs while maintaining high
image quality and text alignment, leveraging the synergy of its guidance
methods, each indispensable in its own right. AMG also features an innovative
automatic detection system for potential memorization during each step of
inference process, allows selective application of guidance strategies,
minimally interfering with the original sampling process to preserve output
utility. We applied AMG to pretrained Denoising Diffusion Probabilistic Models
(DDPM) and Stable Diffusion across various generation tasks. The results
demonstrate that AMG is the first approach to successfully eradicates all
instances of memorization with no or marginal impacts on image quality and
text-alignment, as evidenced by FID and CLIP scores.