Deep Generative Models through the Lens of the Manifold Hypothesis: A
  Survey and New Connections
Gabriel Loaiza-Ganem Brendan Leigh Ross Rasa Hosseinzadeh Anthony L. Caterini Jesse C. Cresswell
In recent years there has been increased interest in understanding the
interplay between deep generative models (DGMs) and the manifold hypothesis.
Research in this area focuses on understanding the reasons why commonly-used
DGMs succeed or fail at learning distributions supported on unknown
low-dimensional manifolds, as well as developing new models explicitly designed
to account for manifold-supported data. This manifold lens provides both
clarity as to why some DGMs (e.g. diffusion models and some generative
adversarial networks) empirically surpass others (e.g. likelihood-based models
such as variational autoencoders, normalizing flows, or energy-based models) at
sample generation, and guidance for devising more performant DGMs. We carry out
the first survey of DGMs viewed through this lens, making two novel
contributions along the way. First, we formally establish that numerical
instability of high-dimensional likelihoods is unavoidable when modelling
low-dimensional data. We then show that DGMs on learned representations of
autoencoders can be interpreted as approximately minimizing Wasserstein
distance: this result, which applies to latent diffusion models, helps justify
their outstanding empirical results. The manifold lens provides a rich
perspective from which to understand DGMs, which we aim to make more accessible
and widespread.