Magic Clothing: Controllable Garment-Driven Image Synthesis
Weifeng Chen Tao Gu Yuhao Xu Chengcai Chen
We propose Magic Clothing, a latent diffusion model (LDM)-based network
architecture for an unexplored garment-driven image synthesis task. Aiming at
generating customized characters wearing the target garments with diverse text
prompts, the image controllability is the most critical issue, i.e., to
preserve the garment details and maintain faithfulness to the text prompts. To
this end, we introduce a garment extractor to capture the detailed garment
features, and employ self-attention fusion to incorporate them into the
pretrained LDMs, ensuring that the garment details remain unchanged on the
target character. Then, we leverage the joint classifier-free guidance to
balance the control of garment features and text prompts over the generated
results. Meanwhile, the proposed garment extractor is a plug-in module
applicable to various finetuned LDMs, and it can be combined with other
extensions like ControlNet and IP-Adapter to enhance the diversity and
controllability of the generated characters. Furthermore, we design
Matched-Points-LPIPS (MP-LPIPS), a robust metric for evaluating the consistency
of the target image to the source garment. Extensive experiments demonstrate
that our Magic Clothing achieves state-of-the-art results under various
conditional controls for garment-driven image synthesis. Our source code is
available at https://github.com/ShineChen1024/MagicClothing.