SODA: Bottleneck Diffusion Models for Representation Learning
Drew A. Hudson Daniel Zoran Mateusz Malinowski Andrew K. Lampinen Andrew Jaegle James L. McClelland Loic Matthey Felix Hill Alexander Lerchner
We introduce SODA, a self-supervised diffusion model, designed for
representation learning. The model incorporates an image encoder, which
distills a source view into a compact representation, that, in turn, guides the
generation of related novel views. We show that by imposing a tight bottleneck
between the encoder and a denoising decoder, and leveraging novel view
synthesis as a self-supervised objective, we can turn diffusion models into
strong representation learners, capable of capturing visual semantics in an
unsupervised manner. To the best of our knowledge, SODA is the first diffusion
model to succeed at ImageNet linear-probe classification, and, at the same
time, it accomplishes reconstruction, editing and synthesis tasks across a wide
range of datasets. Further investigation reveals the disentangled nature of its
emergent latent space, that serves as an effective interface to control and
manipulate the model's produced images. All in all, we aim to shed light on the
exciting and promising potential of diffusion models, not only for image
generation, but also for learning rich and robust representations.