Generalized Contrastive Divergence: Joint Training of Energy-Based Model
  and Diffusion Model through Inverse Reinforcement Learning
Sangwoong Yoon Dohyun Kwon Himchan Hwang Yung-Kyun Noh Frank C. Park
We present Generalized Contrastive Divergence (GCD), a novel objective
function for training an energy-based model (EBM) and a sampler simultaneously.
GCD generalizes Contrastive Divergence (Hinton, 2002), a celebrated algorithm
for training EBM, by replacing Markov Chain Monte Carlo (MCMC) distribution
with a trainable sampler, such as a diffusion model. In GCD, the joint training
of EBM and a diffusion model is formulated as a minimax problem, which reaches
an equilibrium when both models converge to the data distribution. The minimax
learning with GCD bears interesting equivalence to inverse reinforcement
learning, where the energy corresponds to a negative reward, the diffusion
model is a policy, and the real data is expert demonstrations. We present
preliminary yet promising results showing that joint training is beneficial for
both EBM and a diffusion model. GCD enables EBM training without MCMC while
improving the sample quality of a diffusion model.