Correcting Diffusion Generation through Resampling
Yujian Liu Yang Zhang Tommi Jaakkola Shiyu Chang
Despite diffusion models' superior capabilities in modeling complex
distributions, there are still non-trivial distributional discrepancies between
generated and ground-truth images, which has resulted in several notable
problems in image generation, including missing object errors in text-to-image
generation and low image quality. Existing methods that attempt to address
these problems mostly do not tend to address the fundamental cause behind these
problems, which is the distributional discrepancies, and hence achieve
sub-optimal results. In this paper, we propose a particle filtering framework
that can effectively address both problems by explicitly reducing the
distributional discrepancies. Specifically, our method relies on a set of
external guidance, including a small set of real images and a pre-trained
object detector, to gauge the distribution gap, and then design the resampling
weight accordingly to correct the gap. Experiments show that our methods can
effectively correct missing object errors and improve image quality in various
image generation tasks. Notably, our method outperforms the existing strongest
baseline by 5% in object occurrence and 1.0 in FID on MS-COCO. Our code is
publicly available at
https://github.com/UCSB-NLP-Chang/diffusion_resampling.git.