Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass
  Diffusion Transformers
Katherine Crowson Stefan Andreas Baumann Alex Birch Tanishq Mathew Abraham Daniel Z. Kaplan Enrico Shippole
We present the Hourglass Diffusion Transformer (HDiT), an image generative
model that exhibits linear scaling with pixel count, supporting training at
high-resolution (e.g. $1024 \times 1024$) directly in pixel-space. Building on
the Transformer architecture, which is known to scale to billions of
parameters, it bridges the gap between the efficiency of convolutional U-Nets
and the scalability of Transformers. HDiT trains successfully without typical
high-resolution training techniques such as multiscale architectures, latent
autoencoders or self-conditioning. We demonstrate that HDiT performs
competitively with existing models on ImageNet $256^2$, and sets a new
state-of-the-art for diffusion models on FFHQ-$1024^2$.