Fine-Tuning InstructPix2Pix for Advanced Image Colorization
Zifeng An Zijing Xu Eric Fan Qi Cao
This paper presents a novel approach to human image colorization by
fine-tuning the InstructPix2Pix model, which integrates a language model
(GPT-3) with a text-to-image model (Stable Diffusion). Despite the original
InstructPix2Pix model's proficiency in editing images based on textual
instructions, it exhibits limitations in the focused domain of colorization. To
address this, we fine-tuned the model using the IMDB-WIKI dataset, pairing
black-and-white images with a diverse set of colorization prompts generated by
ChatGPT. This paper contributes by (1) applying fine-tuning techniques to
stable diffusion models specifically for colorization tasks, and (2) employing
generative models to create varied conditioning prompts. After finetuning, our
model outperforms the original InstructPix2Pix model on multiple metrics
quantitatively, and we produce more realistically colored images qualitatively.
The code for this project is provided on the GitHub Repository
https://github.com/AllenAnZifeng/DeepLearning282.