Cones 2: Customizable Image Synthesis with Multiple Subjects
Zhiheng Liu Yifei Zhang Yujun Shen Kecheng Zheng Kai Zhu Ruili Feng Yu Liu Deli Zhao Jingren Zhou Yang Cao
Synthesizing images with user-specified subjects has received growing
attention due to its practical applications. Despite the recent success in
single subject customization, existing algorithms suffer from high training
cost and low success rate along with increased number of subjects. Towards
controllable image synthesis with multiple subjects as the constraints, this
work studies how to efficiently represent a particular subject as well as how
to appropriately compose different subjects. We find that the text embedding
regarding the subject token already serves as a simple yet effective
representation that supports arbitrary combinations without any model tuning.
Through learning a residual on top of the base embedding, we manage to robustly
shift the raw subject to the customized subject given various text conditions.
We then propose to employ layout, a very abstract and easy-to-obtain prior, as
the spatial guidance for subject arrangement. By rectifying the activations in
the cross-attention map, the layout appoints and separates the location of
different subjects in the image, significantly alleviating the interference
across them. Both qualitative and quantitative experimental results demonstrate
our superiority over state-of-the-art alternatives under a variety of settings
for multi-subject customization.