Factorized Diffusion: Perceptual Illusions by Noise Decomposition
Daniel Geng Inbum Park Andrew Owens
Given a factorization of an image into a sum of linear components, we present
a zero-shot method to control each individual component through diffusion model
sampling. For example, we can decompose an image into low and high spatial
frequencies and condition these components on different text prompts. This
produces hybrid images, which change appearance depending on viewing distance.
By decomposing an image into three frequency subbands, we can generate hybrid
images with three prompts. We also use a decomposition into grayscale and color
components to produce images whose appearance changes when they are viewed in
grayscale, a phenomena that naturally occurs under dim lighting. And we explore
a decomposition by a motion blur kernel, which produces images that change
appearance under motion blurring. Our method works by denoising with a
composite noise estimate, built from the components of noise estimates
conditioned on different prompts. We also show that for certain decompositions,
our method recovers prior approaches to compositional generation and spatial
control. Finally, we show that we can extend our approach to generate hybrid
images from real images. We do this by holding one component fixed and
generating the remaining components, effectively solving an inverse problem.