U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models
Ilias Mitsouras Eleftherios Tsonis Paraskevi Tzouveli Athanasios Voulodimos
Diffusion models have demonstrated remarkable performance in text-to-image
synthesis, producing realistic and high resolution images that faithfully
adhere to the corresponding text-prompts. Despite their great success, they
still fall behind in sketch-to-image synthesis tasks, where in addition to
text-prompts, the spatial layout of the generated images has to closely follow
the outlines of certain reference sketches. Employing an MLP latent edge
predictor to guide the spatial layout of the synthesized image by predicting
edge maps at each denoising step has been recently proposed. Despite yielding
promising results, the pixel-wise operation of the MLP does not take into
account the spatial layout as a whole, and demands numerous denoising
iterations to produce satisfactory images, leading to time inefficiency. To
this end, we introduce U-Sketch, a framework featuring a U-Net type latent edge
predictor, which is capable of efficiently capturing both local and global
features, as well as spatial correlations between pixels. Moreover, we propose
the addition of a sketch simplification network that offers the user the choice
of preprocessing and simplifying input sketches for enhanced outputs. The
experimental results, corroborated by user feedback, demonstrate that our
proposed U-Net latent edge predictor leads to more realistic results, that are
better aligned with the spatial outlines of the reference sketches, while
drastically reducing the number of required denoising steps and, consequently,
the overall execution time.