ControlStyle: Text-Driven Stylized Image Generation Using Diffusion
  Priors
Jingwen Chen Yingwei Pan Ting Yao Tao Mei
Recently, the multimedia community has witnessed the rise of diffusion models
trained on large-scale multi-modal data for visual content creation,
particularly in the field of text-to-image generation. In this paper, we
propose a new task for ``stylizing'' text-to-image models, namely text-driven
stylized image generation, that further enhances editability in content
creation. Given input text prompt and style image, this task aims to produce
stylized images which are both semantically relevant to input text prompt and
meanwhile aligned with the style image in style. To achieve this, we present a
new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image
model with a trainable modulation network enabling more conditions of text
prompts and style images. Moreover, diffusion style and content regularizations
are simultaneously introduced to facilitate the learning of this modulation
network with these diffusion priors, pursuing high-quality stylized
text-to-image generation. Extensive experiments demonstrate the effectiveness
of our ControlStyle in producing more visually pleasing and artistic results,
surpassing a simple combination of text-to-image model and conventional style
transfer techniques.