FT-Shield: A Watermark Against Unauthorized Fine-tuning in Text-to-Image
  Diffusion Models
Yingqian Cui Jie Ren Yuping Lin Han Xu Pengfei He Yue Xing Wenqi Fan Hui Liu Jiliang Tang
Text-to-image generative models based on latent diffusion models (LDM) have
demonstrated their outstanding ability in generating high-quality and
high-resolution images according to language prompt. Based on these powerful
latent diffusion models, various fine-tuning methods have been proposed to
achieve the personalization of text-to-image diffusion models such as artistic
style adaptation and human face transfer. However, the unauthorized usage of
data for model personalization has emerged as a prevalent concern in relation
to copyright violations. For example, a malicious user may use the fine-tuning
technique to generate images which mimic the style of a painter without his/her
permission. In light of this concern, we have proposed FT-Shield, a
watermarking approach specifically designed for the fine-tuning of
text-to-image diffusion models to aid in detecting instances of infringement.
We develop a novel algorithm for the generation of the watermark to ensure that
the watermark on the training images can be quickly and accurately transferred
to the generated images of text-to-image diffusion models. A watermark will be
detected on an image by a binary watermark detector if the image is generated
by a model that has been fine-tuned using the protected watermarked images.
Comprehensive experiments were conducted to validate the effectiveness of
FT-Shield.