Implicit Style-Content Separation using B-LoRA
Yarden Frenkel Yael Vinker Ariel Shamir Daniel Cohen-Or
Image stylization involves manipulating the visual appearance and texture
(style) of an image while preserving its underlying objects, structures, and
concepts (content). The separation of style and content is essential for
manipulating the image's style independently from its content, ensuring a
harmonious and visually pleasing result. Achieving this separation requires a
deep understanding of both the visual and semantic characteristics of images,
often necessitating the training of specialized models or employing heavy
optimization. In this paper, we introduce B-LoRA, a method that leverages LoRA
(Low-Rank Adaptation) to implicitly separate the style and content components
of a single image, facilitating various image stylization tasks. By analyzing
the architecture of SDXL combined with LoRA, we find that jointly learning the
LoRA weights of two specific blocks (referred to as B-LoRAs) achieves
style-content separation that cannot be achieved by training each B-LoRA
independently. Consolidating the training into only two blocks and separating
style and content allows for significantly improving style manipulation and
overcoming overfitting issues often associated with model fine-tuning. Once
trained, the two B-LoRAs can be used as independent components to allow various
image stylization tasks, including image style transfer, text-based image
stylization, consistent style generation, and style-content mixing.