ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and
  Implicit Style Prompt Bank
Zhanjie Zhang Quanwei Zhang Guangyuan Li Wei Xing Lei Zhao Jiakai Sun Zehua Lan Junsheng Luan Yiling Huang Huaizhong Lin
Artistic style transfer aims to repaint the content image with the learned
artistic style. Existing artistic style transfer methods can be divided into
two categories: small model-based approaches and pre-trained large-scale
model-based approaches. Small model-based approaches can preserve the content
strucuture, but fail to produce highly realistic stylized images and introduce
artifacts and disharmonious patterns; Pre-trained large-scale model-based
approaches can generate highly realistic stylized images but struggle with
preserving the content structure. To address the above issues, we propose
ArtBank, a novel artistic style transfer framework, to generate highly
realistic stylized images while preserving the content structure of the content
images. Specifically, to sufficiently dig out the knowledge embedded in
pre-trained large-scale models, an Implicit Style Prompt Bank (ISPB), a set of
trainable parameter matrices, is designed to learn and store knowledge from the
collection of artworks and behave as a visual prompt to guide pre-trained
large-scale models to generate highly realistic stylized images while
preserving content structure. Besides, to accelerate training the above ISPB,
we propose a novel Spatial-Statistical-based self-Attention Module (SSAM). The
qualitative and quantitative experiments demonstrate the superiority of our
proposed method over state-of-the-art artistic style transfer methods.