BinaryDM: Towards Accurate Binarization of Diffusion Model
Xingyu Zheng Haotong Qin Xudong Ma Mingyuan Zhang Haojie Hao Jiakai Wang Zixiang Zhao Jinyang Guo Xianglong Liu
With the advancement of diffusion models (DMs) and the substantially
increased computational requirements, quantization emerges as a practical
solution to obtain compact and efficient low-bit DMs. However, the highly
discrete representation leads to severe accuracy degradation, hindering the
quantization of diffusion models to ultra-low bit-widths. In this paper, we
propose BinaryDM, a novel accurate quantization-aware training approach to push
the weights of diffusion models towards the limit of 1-bit. Firstly, we present
a Learnable Multi-basis Binarizer (LMB) to recover the representations
generated by the binarized DM, which improves the information in details of
representations crucial to the DM. Secondly, a Low-rank Representation
Mimicking (LRM) is applied to enhance the binarization-aware optimization of
the DM, alleviating the optimization direction ambiguity caused by fine-grained
alignment. Moreover, a progressive initialization strategy is applied to
training DMs to avoid convergence difficulties. Comprehensive experiments
demonstrate that BinaryDM achieves significant accuracy and efficiency gains
compared to SOTA quantization methods of DMs under ultra-low bit-widths. As the
first binarization method for diffusion models, BinaryDM achieves impressive
16.0 times FLOPs and 27.1 times storage savings with 1-bit weight and 4-bit
activation, showcasing its substantial advantages and potential for deploying
DMs on resource-limited scenarios.