STANLEY: Stochastic Gradient Anisotropic Langevin Dynamics for Learning
  Energy-Based Models
Belhal Karimi Jianwen Xie Ping Li
We propose in this paper, STANLEY, a STochastic gradient ANisotropic LangEvin
dYnamics, for sampling high dimensional data. With the growing efficacy and
potential of Energy-Based modeling, also known as non-normalized probabilistic
modeling, for modeling a generative process of different natures of high
dimensional data observations, we present an end-to-end learning algorithm for
Energy-Based models (EBM) with the purpose of improving the quality of the
resulting sampled data points. While the unknown normalizing constant of EBMs
makes the training procedure intractable, resorting to Markov Chain Monte Carlo
(MCMC) is in general a viable option. Realizing what MCMC entails for the EBM
training, we propose in this paper, a novel high dimensional sampling method,
based on an anisotropic stepsize and a gradient-informed covariance matrix,
embedded into a discretized Langevin diffusion. We motivate the necessity for
an anisotropic update of the negative samples in the Markov Chain by the
nonlinearity of the backbone of the EBM, here a Convolutional Neural Network.
Our resulting method, namely STANLEY, is an optimization algorithm for training
Energy-Based models via our newly introduced MCMC method. We provide a
theoretical understanding of our sampling scheme by proving that the sampler
leads to a geometrically uniformly ergodic Markov Chain. Several image
generation experiments are provided in our paper to show the effectiveness of
our method.