TokenCompose: Grounding Diffusion with Token-level Supervision
Zirui Wang Zhizhou Sha Zheng Ding Yilin Wang Zhuowen Tu
We present TokenCompose, a Latent Diffusion Model for text-to-image
generation that achieves enhanced consistency between user-specified text
prompts and model-generated images. Despite its tremendous success, the
standard denoising process in the Latent Diffusion Model takes text prompts as
conditions only, absent explicit constraint for the consistency between the
text prompts and the image contents, leading to unsatisfactory results for
composing multiple object categories. TokenCompose aims to improve
multi-category instance composition by introducing the token-wise consistency
terms between the image content and object segmentation maps in the finetuning
stage. TokenCompose can be applied directly to the existing training pipeline
of text-conditioned diffusion models without extra human labeling information.
By finetuning Stable Diffusion, the model exhibits significant improvements in
multi-category instance composition and enhanced photorealism for its generated
images.