LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept
  Customization in Training-Free Diffusion Models
Yang Yang Wen Wang Liang Peng Chaotian Song Yao Chen Hengjia Li Xiaolong Yang Qinglin Lu Deng Cai Boxi Wu Wei Liu
Customization generation techniques have significantly advanced the synthesis
of specific concepts across varied contexts. Multi-concept customization
emerges as the challenging task within this domain. Existing approaches often
rely on training a Low-Rank Adaptations (LoRA) fusion matrix of multiple LoRA
to merge various concepts into a single image. However, we identify this
straightforward method faces two major challenges: 1) concept confusion, which
occurs when the model cannot preserve distinct individual characteristics, and
2) concept vanishing, where the model fails to generate the intended subjects.
To address these issues, we introduce LoRA-Composer, a training-free framework
designed for seamlessly integrating multiple LoRAs, thereby enhancing the
harmony among different concepts within generated images. LoRA-Composer
addresses concept vanishing through Concept Injection Constraints, enhancing
concept visibility via an expanded cross-attention mechanism. To combat concept
confusion, Concept Isolation Constraints are introduced, refining the
self-attention computation. Furthermore, Latent Re-initialization is proposed
to effectively stimulate concept-specific latent within designated regions. Our
extensive testing showcases a notable enhancement in LoRA-Composer's
performance compared to standard baselines, especially when eliminating the
image-based conditions like canny edge or pose estimations. Code is released at
https://github.com/Young98CN/LoRA\_Composer.