Removing Undesirable Concepts in Text-to-Image Generative Models with
  Learnable Prompts
Anh Bui Khanh Doan Trung Le Paul Montague Tamas Abraham Dinh Phung
Generative models have demonstrated remarkable potential in generating
visually impressive content from textual descriptions. However, training these
models on unfiltered internet data poses the risk of learning and subsequently
propagating undesirable concepts, such as copyrighted or unethical content. In
this paper, we propose a novel method to remove undesirable concepts from
text-to-image generative models by incorporating a learnable prompt into the
cross-attention module. This learnable prompt acts as additional memory to
transfer the knowledge of undesirable concepts into it and reduce the
dependency of these concepts on the model parameters and corresponding textual
inputs. Because of this knowledge transfer into the prompt, erasing these
undesirable concepts is more stable and has minimal negative impact on other
concepts. We demonstrate the effectiveness of our method on the Stable
Diffusion model, showcasing its superiority over state-of-the-art erasure
methods in terms of removing undesirable content while preserving other
unrelated elements.