DeeDiff: Dynamic Uncertainty-Aware Early Exiting for Accelerating
  Diffusion Model Generation
Shengkun Tang Yaqing Wang Caiwen Ding Yi Liang Yao Li Dongkuan Xu
Diffusion models achieve great success in generating diverse and
high-fidelity images. The performance improvements come with low generation
speed per image, which hinders the application diffusion models in real-time
scenarios. While some certain predictions benefit from the full computation of
the model in each sample iteration, not every iteration requires the same
amount of computation, potentially leading to computation waste. In this work,
we propose DeeDiff, an early exiting framework that adaptively allocates
computation resources in each sampling step to improve the generation
efficiency of diffusion models. Specifically, we introduce a timestep-aware
uncertainty estimation module (UEM) for diffusion models which is attached to
each intermediate layer to estimate the prediction uncertainty of each layer.
The uncertainty is regarded as the signal to decide if the inference
terminates. Moreover, we propose uncertainty-aware layer-wise loss to fill the
performance gap between full models and early-exited models. With such loss
strategy, our model is able to obtain comparable results as full-layer models.
Extensive experiments of class-conditional, unconditional, and text-guided
generation on several datasets show that our method achieves state-of-the-art
performance and efficiency trade-off compared with existing early exiting
methods on diffusion models. More importantly, our method even brings extra
benefits to baseline models and obtains better performance on CIFAR-10 and
Celeb-A datasets. Full code and model are released for reproduction.