End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive
  Divergence with Local Mode Initialization
Shohei Taniguchi Masahiro Suzuki Yusuke Iwasawa Yutaka Matsuo
We address the problem of biased gradient estimation in deep Boltzmann
machines (DBMs). The existing method to obtain an unbiased estimator uses a
maximal coupling based on a Gibbs sampler, but when the state is
high-dimensional, it takes a long time to converge. In this study, we propose
to use a coupling based on the Metropolis-Hastings (MH) and to initialize the
state around a local mode of the target distribution. Because of the propensity
of MH to reject proposals, the coupling tends to converge in only one step with
a high probability, leading to high efficiency. We find that our method allows
DBMs to be trained in an end-to-end fashion without greedy pretraining. We also
propose some practical techniques to further improve the performance of DBMs.
We empirically demonstrate that our training algorithm enables DBMs to show
comparable generative performance to other deep generative models, achieving
the FID score of 10.33 for MNIST.