SD-DiT: Unleashing the Power of Self-supervised Discrimination in
  Diffusion Transformer
Rui Zhu Yingwei Pan Yehao Li Ting Yao Zhenglong Sun Tao Mei Chang Wen Chen
Diffusion Transformer (DiT) has emerged as the new trend of generative
diffusion models on image generation. In view of extremely slow convergence in
typical DiT, recent breakthroughs have been driven by mask strategy that
significantly improves the training efficiency of DiT with additional
intra-image contextual learning. Despite this progress, mask strategy still
suffers from two inherent limitations: (a) training-inference discrepancy and
(b) fuzzy relations between mask reconstruction & generative diffusion process,
resulting in sub-optimal training of DiT. In this work, we address these
limitations by novelly unleashing the self-supervised discrimination knowledge
to boost DiT training. Technically, we frame our DiT in a teacher-student
manner. The teacher-student discriminative pairs are built on the diffusion
noises along the same Probability Flow Ordinary Differential Equation (PF-ODE).
Instead of applying mask reconstruction loss over both DiT encoder and decoder,
we decouple DiT encoder and decoder to separately tackle discriminative and
generative objectives. In particular, by encoding discriminative pairs with
student and teacher DiT encoders, a new discriminative loss is designed to
encourage the inter-image alignment in the self-supervised embedding space.
After that, student samples are fed into student DiT decoder to perform the
typical generative diffusion task. Extensive experiments are conducted on
ImageNet dataset, and our method achieves a competitive balance between
training cost and generative capacity.