Block and Detail: Scaffolding Sketch-to-Image Generation
Vishnu Sarukkai Lu Yuan Mia Tang Maneesh Agrawala Kayvon Fatahalian
We introduce a novel sketch-to-image tool that aligns with the iterative
refinement process of artists. Our tool lets users sketch blocking strokes to
coarsely represent the placement and form of objects and detail strokes to
refine their shape and silhouettes. We develop a two-pass algorithm for
generating high-fidelity images from such sketches at any point in the
iterative process. In the first pass we use a ControlNet to generate an image
that strictly follows all the strokes (blocking and detail) and in the second
pass we add variation by renoising regions surrounding blocking strokes. We
also present a dataset generation scheme that, when used to train a ControlNet
architecture, allows regions that do not contain strokes to be interpreted as
not-yet-specified regions rather than empty space. We show that this
partial-sketch-aware ControlNet can generate coherent elements from partial
sketches that only contain a small number of strokes. The high-fidelity images
produced by our approach serve as scaffolds that can help the user adjust the
shape and proportions of objects or add additional elements to the composition.
We demonstrate the effectiveness of our approach with a variety of examples and
evaluative comparisons.