Diffusion Model with Cross Attention as an Inductive Bias for
  Disentanglement
Tao Yang Cuiling Lan Yan Lu Nanning zheng
Disentangled representation learning strives to extract the intrinsic factors
within observed data. Factorizing these representations in an unsupervised
manner is notably challenging and usually requires tailored loss functions or
specific structural designs. In this paper, we introduce a new perspective and
framework, demonstrating that diffusion models with cross-attention can serve
as a powerful inductive bias to facilitate the learning of disentangled
representations. We propose to encode an image to a set of concept tokens and
treat them as the condition of the latent diffusion for image reconstruction,
where cross-attention over the concept tokens is used to bridge the interaction
between the encoder and diffusion. Without any additional regularization, this
framework achieves superior disentanglement performance on the benchmark
datasets, surpassing all previous methods with intricate designs. We have
conducted comprehensive ablation studies and visualization analysis, shedding
light on the functioning of this model. This is the first work to reveal the
potent disentanglement capability of diffusion models with cross-attention,
requiring no complex designs. We anticipate that our findings will inspire more
investigation on exploring diffusion for disentangled representation learning
towards more sophisticated data analysis and understanding.