Diffusing Colors: Image Colorization with Text Guided Diffusion
Nir Zabari Aharon Azulay Alexey Gorkor Tavi Halperin Ohad Fried
The colorization of grayscale images is a complex and subjective task with
significant challenges. Despite recent progress in employing large-scale
datasets with deep neural networks, difficulties with controllability and
visual quality persist. To tackle these issues, we present a novel image
colorization framework that utilizes image diffusion techniques with granular
text prompts. This integration not only produces colorization outputs that are
semantically appropriate but also greatly improves the level of control users
have over the colorization process. Our method provides a balance between
automation and control, outperforming existing techniques in terms of visual
quality and semantic coherence. We leverage a pretrained generative Diffusion
Model, and show that we can finetune it for the colorization task without
losing its generative power or attention to text prompts. Moreover, we present
a novel CLIP-based ranking model that evaluates color vividness, enabling
automatic selection of the most suitable level of vividness based on the
specific scene semantics. Our approach holds potential particularly for color
enhancement and historical image colorization.