Elucidating The Design Space of Classifier-Guided Diffusion Generation
Jiajun Ma Tianyang Hu Wenjia Wang Jiacheng Sun
Guidance in conditional diffusion generation is of great importance for
sample quality and controllability. However, existing guidance schemes are to
be desired. On one hand, mainstream methods such as classifier guidance and
classifier-free guidance both require extra training with labeled data, which
is time-consuming and unable to adapt to new conditions. On the other hand,
training-free methods such as universal guidance, though more flexible, have
yet to demonstrate comparable performance. In this work, through a
comprehensive investigation into the design space, we show that it is possible
to achieve significant performance improvements over existing guidance schemes
by leveraging off-the-shelf classifiers in a training-free fashion, enjoying
the best of both worlds. Employing calibration as a general guideline, we
propose several pre-conditioning techniques to better exploit pretrained
off-the-shelf classifiers for guiding diffusion generation. Extensive
experiments on ImageNet validate our proposed method, showing that
state-of-the-art diffusion models (DDPM, EDM, DiT) can be further improved (up
to 20%) using off-the-shelf classifiers with barely any extra computational
cost. With the proliferation of publicly available pretrained classifiers, our
proposed approach has great potential and can be readily scaled up to
text-to-image generation tasks. The code is available at
https://github.com/AlexMaOLS/EluCD/tree/main.