Memory-Efficient Personalization using Quantized Diffusion Model
Hyogon Ryu Seohyun Lim Hyunjung Shim
The rise of billion-parameter diffusion models like Stable Diffusion XL,
Imagen, and Dall-E3 markedly advances the field of generative AI. However,
their large-scale nature poses challenges in fine-tuning and deployment due to
high resource demands and slow inference speed. This paper ventures into the
relatively unexplored yet promising realm of fine-tuning quantized diffusion
models. We establish a strong baseline by customizing three models: PEQA for
fine-tuning quantization parameters, Q-Diffusion for post-training
quantization, and DreamBooth for personalization. Our analysis reveals a
notable trade-off between subject and prompt fidelity within the baseline
model. To address these issues, we introduce two strategies, inspired by the
distinct roles of different timesteps in diffusion models: S1 optimizing a
single set of fine-tuning parameters exclusively at selected intervals, and S2
creating multiple fine-tuning parameter sets, each specialized for different
timestep intervals. Our approach not only enhances personalization but also
upholds prompt fidelity and image quality, significantly outperforming the
baseline qualitatively and quantitatively. The code will be made publicly
available.