Learning Hierarchical Features with Joint Latent Space Energy-Based
  Prior
Jiali Cui Ying Nian Wu Tian Han
This paper studies the fundamental problem of multi-layer generator models in
learning hierarchical representations. The multi-layer generator model that
consists of multiple layers of latent variables organized in a top-down
architecture tends to learn multiple levels of data abstraction. However, such
multi-layer latent variables are typically parameterized to be Gaussian, which
can be less informative in capturing complex abstractions, resulting in limited
success in hierarchical representation learning. On the other hand, the
energy-based (EBM) prior is known to be expressive in capturing the data
regularities, but it often lacks the hierarchical structure to capture
different levels of hierarchical representations. In this paper, we propose a
joint latent space EBM prior model with multi-layer latent variables for
effective hierarchical representation learning. We develop a variational joint
learning scheme that seamlessly integrates an inference model for efficient
inference. Our experiments demonstrate that the proposed joint EBM prior is
effective and expressive in capturing hierarchical representations and
modelling data distribution.