Phasic Content Fusing Diffusion Model with Directional Distribution
  Consistency for Few-Shot Model Adaption
Teng Hu Jiangning Zhang Liang Liu Ran Yi Siqi Kou Haokun Zhu Xu Chen Yabiao Wang Chengjie Wang Lizhuang Ma
Training a generative model with limited number of samples is a challenging
task. Current methods primarily rely on few-shot model adaption to train the
network. However, in scenarios where data is extremely limited (less than 10),
the generative network tends to overfit and suffers from content degradation.
To address these problems, we propose a novel phasic content fusing few-shot
diffusion model with directional distribution consistency loss, which targets
different learning objectives at distinct training stages of the diffusion
model. Specifically, we design a phasic training strategy with phasic content
fusion to help our model learn content and style information when t is large,
and learn local details of target domain when t is small, leading to an
improvement in the capture of content, style and local details. Furthermore, we
introduce a novel directional distribution consistency loss that ensures the
consistency between the generated and source distributions more efficiently and
stably than the prior methods, preventing our model from overfitting. Finally,
we propose a cross-domain structure guidance strategy that enhances structure
consistency during domain adaptation. Theoretical analysis, qualitative and
quantitative experiments demonstrate the superiority of our approach in
few-shot generative model adaption tasks compared to state-of-the-art methods.
The source code is available at:
https://github.com/sjtuplayer/few-shot-diffusion.