ReFACT: Updating Text-to-Image Models by Editing the Text Encoder
Dana Arad Hadas Orgad Yonatan Belinkov
Text-to-image models are trained on extensive amounts of data, leading them
to implicitly encode factual knowledge within their parameters. While some
facts are useful, others may be incorrect or become outdated (e.g., the current
President of the United States). We introduce ReFACT, a novel approach for
editing factual knowledge in text-to-image generative models. ReFACT updates
the weights of a specific layer in the text encoder, only modifying a tiny
portion of the model's parameters, and leaving the rest of the model
unaffected. We empirically evaluate ReFACT on an existing benchmark, alongside
RoAD, a newly curated dataset. ReFACT achieves superior performance in terms of
generalization to related concepts while preserving unrelated concepts.
Furthermore, ReFACT maintains image generation quality, making it a valuable
tool for updating and correcting factual information in text-to-image models.