DiffCloth: Diffusion Based Garment Synthesis and Manipulation via
  Structural Cross-modal Semantic Alignment
Xujie Zhang Binbin Yang Michael C. Kampffmeyer Wenqing Zhang Shiyue Zhang Guansong Lu Liang Lin Hang Xu Xiaodan Liang
Cross-modal garment synthesis and manipulation will significantly benefit the
way fashion designers generate garments and modify their designs via flexible
linguistic interfaces.Current approaches follow the general text-to-image
paradigm and mine cross-modal relations via simple cross-attention modules,
neglecting the structural correspondence between visual and textual
representations in the fashion design domain. In this work, we instead
introduce DiffCloth, a diffusion-based pipeline for cross-modal garment
synthesis and manipulation, which empowers diffusion models with flexible
compositionality in the fashion domain by structurally aligning the cross-modal
semantics. Specifically, we formulate the part-level cross-modal alignment as a
bipartite matching problem between the linguistic Attribute-Phrases (AP) and
the visual garment parts which are obtained via constituency parsing and
semantic segmentation, respectively. To mitigate the issue of attribute
confusion, we further propose a semantic-bundled cross-attention to preserve
the spatial structure similarities between the attention maps of attribute
adjectives and part nouns in each AP. Moreover, DiffCloth allows for
manipulation of the generated results by simply replacing APs in the text
prompts. The manipulation-irrelevant regions are recognized by blended masks
obtained from the bundled attention maps of the APs and kept unchanged.
Extensive experiments on the CM-Fashion benchmark demonstrate that DiffCloth
both yields state-of-the-art garment synthesis results by leveraging the
inherent structural information and supports flexible manipulation with region
consistency.