Diffusion Models Without Attention
Jing Nathan Yan Jiatao Gu Alexander M. Rush
In recent advancements in high-fidelity image generation, Denoising Diffusion
Probabilistic Models (DDPMs) have emerged as a key player. However, their
application at high resolutions presents significant computational challenges.
Current methods, such as patchifying, expedite processes in UNet and
Transformer architectures but at the expense of representational capacity.
Addressing this, we introduce the Diffusion State Space Model (DiffuSSM), an
architecture that supplants attention mechanisms with a more scalable state
space model backbone. This approach effectively handles higher resolutions
without resorting to global compression, thus preserving detailed image
representation throughout the diffusion process. Our focus on FLOP-efficient
architectures in diffusion training marks a significant step forward.
Comprehensive evaluations on both ImageNet and LSUN datasets at two resolutions
demonstrate that DiffuSSMs are on par or even outperform existing diffusion
models with attention modules in FID and Inception Score metrics while
significantly reducing total FLOP usage.