MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask
Yupeng Zhou Daquan Zhou Zuo-Liang Zhu Yaxing Wang Qibin Hou Jiashi Feng
Recent advancements in diffusion models have showcased their impressive
capacity to generate visually striking images. Nevertheless, ensuring a close
match between the generated image and the given prompt remains a persistent
challenge. In this work, we identify that a crucial factor leading to the
text-image mismatch issue is the inadequate cross-modality relation learning
between the prompt and the output image. To better align the prompt and image
content, we advance the cross-attention with an adaptive mask, which is
conditioned on the attention maps and the prompt embeddings, to dynamically
adjust the contribution of each text token to the image features. This
mechanism explicitly diminishes the ambiguity in semantic information embedding
from the text encoder, leading to a boost of text-to-image consistency in the
synthesized images. Our method, termed MaskDiffusion, is training-free and
hot-pluggable for popular pre-trained diffusion models. When applied to the
latent diffusion models, our MaskDiffusion can significantly improve the
text-to-image consistency with negligible computation overhead compared to the
original diffusion models.