SparseDM: Toward Sparse Efficient Diffusion Models
Kafeng Wang Jianfei Chen He Li Zhenpeng Mi Jun Zhu
Diffusion models have been extensively used in data generation tasks and are
recognized as one of the best generative models. However, their time-consuming
deployment, long inference time, and requirements on large memory limit their
application on mobile devices. In this paper, we propose a method based on the
improved Straight-Through Estimator to improve the deployment efficiency of
diffusion models. Specifically, we add sparse masks to the Convolution and
Linear layers in a pre-trained diffusion model, then use design progressive
sparsity for model training in the fine-tuning stage, and switch the inference
mask on and off, which supports a flexible choice of sparsity during inference
according to the FID and MACs requirements. Experiments on four datasets
conducted on a state-of-the-art Transformer-based diffusion model demonstrate
that our method reduces MACs by $50\%$ while increasing FID by only 1.5 on
average. Under other MACs conditions, the FID is also lower than 1$\sim$137
compared to other methods.