HiDiffusion: Unlocking High-Resolution Creativity and Efficiency in
  Low-Resolution Trained Diffusion Models
Shen Zhang Zhaowei Chen Zhenyu Zhao Zhenyuan Chen Yao Tang Yuhao Chen Wengang Cao Jiajun Liang
We introduce HiDiffusion, a tuning-free framework comprised of
Resolution-Aware U-Net (RAU-Net) and Modified Shifted Window Multi-head
Self-Attention (MSW-MSA) to enable pretrained large text-to-image diffusion
models to efficiently generate high-resolution images (e.g. 1024$\times$1024)
that surpass the training image resolution. Pretrained diffusion models
encounter unreasonable object duplication in generating images beyond the
training image resolution. We attribute it to the mismatch between the feature
map size of high-resolution images and the receptive field of U-Net's
convolution. To address this issue, we propose a simple yet scalable method
named RAU-Net. RAU-Net dynamically adjusts the feature map size to match the
convolution's receptive field in the deep block of U-Net. Another obstacle in
high-resolution synthesis is the slow inference speed of U-Net. Our
observations reveal that the global self-attention in the top block, which
exhibits locality, however, consumes the majority of computational resources.
To tackle this issue, we propose MSW-MSA. Unlike previous window attention
mechanisms, our method uses a much larger window size and dynamically shifts
windows to better accommodate diffusion models. Extensive experiments
demonstrate that our HiDiffusion can scale diffusion models to generate
1024$\times$1024, 2048$\times$2048, or even 4096$\times$4096 resolution images,
while simultaneously reducing inference time by 40\%-60\%, achieving
state-of-the-art performance on high-resolution image synthesis. The most
significant revelation of our work is that a pretrained diffusion model on
low-resolution images is scalable for high-resolution generation without
further tuning. We hope this revelation can provide insights for future
research on the scalability of diffusion models.