Image Inpainting via Tractable Steering of Diffusion Models
Anji Liu Mathias Niepert Guy Van den Broeck
Diffusion models are the current state of the art for generating
photorealistic images. Controlling the sampling process for constrained image
generation tasks such as inpainting, however, remains challenging since exact
conditioning on such constraints is intractable. While existing methods use
various techniques to approximate the constrained posterior, this paper
proposes to exploit the ability of Tractable Probabilistic Models (TPMs) to
exactly and efficiently compute the constrained posterior, and to leverage this
signal to steer the denoising process of diffusion models. Specifically, this
paper adopts a class of expressive TPMs termed Probabilistic Circuits (PCs).
Building upon prior advances, we further scale up PCs and make them capable of
guiding the image generation process of diffusion models. Empirical results
suggest that our approach can consistently improve the overall quality and
semantic coherence of inpainted images across three natural image datasets
(i.e., CelebA-HQ, ImageNet, and LSUN) with only ~10% additional computational
overhead brought by the TPM. Further, with the help of an image encoder and
decoder, our method can readily accept semantic constraints on specific regions
of the image, which opens up the potential for more controlled image generation
tasks. In addition to proposing a new framework for constrained image
generation, this paper highlights the benefit of more tractable models and
motivates the development of expressive TPMs.