Frequency-Time Diffusion with Neural Cellular Automata
John Kalkhof Arlene Kühn Yannik Frisch Anirban Mukhopadhyay
Denoising Diffusion Models (DDMs) have become the leading generative
technique for synthesizing high-quality images but are often constrained by
their UNet-based architectures that impose certain limitations. In particular,
the considerable size of often hundreds of millions of parameters makes them
impractical when hardware resources are limited. However, even with powerful
hardware, processing images in the gigapixel range is difficult. This is
especially true in fields such as microscopy or satellite imaging, where such
challenges arise from the limitation to a predefined generative size and the
inefficient scaling to larger images. We present two variations of Neural
Cellular Automata (NCA)-based DDM methods to address these challenges and
jumpstart NCA-based DDMs: Diff-NCA and FourierDiff-NCA. Diff-NCA performs
diffusion by using only local features of the underlying distribution, making
it suitable for applications where local features are critical. To communicate
global knowledge in image space, naive NCA setups require timesteps that
increase with the image scale. We solve this bottleneck of current NCA
architectures by introducing FourierDiff-NCA, which advances Diff-NCA by adding
a Fourier-based diffusion process and combines the frequency-organized Fourier
space with the image space. By initiating diffusion in the Fourier domain and
finalizing it in the image space, FourierDiff-NCA accelerates global
communication. We validate our techniques by using Diff-NCA (208k parameters)
to generate high-resolution digital pathology scans at 576x576 resolution and
FourierDiff-NCA (887k parameters) to synthesize CelebA images at 64x64,
outperforming VNCA and five times bigger UNet-based DDMs. In addition, we
demonstrate FourierDiff-NCA's capabilities in super-resolution, OOD image
synthesis, and inpainting without additional training.