Bring Metric Functions into Diffusion Models
Jie An Zhengyuan Yang Jianfeng Wang Linjie Li Zicheng Liu Lijuan Wang Jiebo Luo
We introduce a Cascaded Diffusion Model (Cas-DM) that improves a Denoising
Diffusion Probabilistic Model (DDPM) by effectively incorporating additional
metric functions in training. Metric functions such as the LPIPS loss have been
proven highly effective in consistency models derived from the score matching.
However, for the diffusion counterparts, the methodology and efficacy of adding
extra metric functions remain unclear. One major challenge is the mismatch
between the noise predicted by a DDPM at each step and the desired clean image
that the metric function works well on. To address this problem, we propose
Cas-DM, a network architecture that cascades two network modules to effectively
apply metric functions to the diffusion model training. The first module,
similar to a standard DDPM, learns to predict the added noise and is unaffected
by the metric function. The second cascaded module learns to predict the clean
image, thereby facilitating the metric function computation. Experiment results
show that the proposed diffusion model backbone enables the effective use of
the LPIPS loss, leading to state-of-the-art image quality (FID, sFID, IS) on
various established benchmarks.