ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image
  and Text
Dingkun Yan Liang Yuan Yuma Nishioka Issei Fujishiro Suguru Saito
Recently, diffusion models have demonstrated their effectiveness in
generating extremely high-quality images and have found wide-ranging
applications, including automatic sketch colorization. However, most existing
models use text to guide the conditional generation, with fewer attempts
exploring the potential advantages of using image tokens as conditional inputs
for networks. As such, this paper exhaustively investigates image-guided
models, specifically targeting reference-based sketch colorization, which aims
to colorize sketch images using reference color images. We investigate three
critical aspects of reference-based diffusion models: the shortcomings compared
to text-based counterparts, the training strategies, and the capability in
zero-shot, sequential text-based manipulation. We introduce two variations of
an image-guided latent diffusion model using different image tokens from the
pre-trained CLIP image encoder, and we propose corresponding manipulation
methods to adjust their results sequentially using weighted text inputs. We
conduct comprehensive evaluations of our models through qualitative and
quantitative experiments, as well as a user study.