Learning Unnormalized Statistical Models via Compositional Optimization
Wei Jiang Jiayu Qin Lingyu Wu Changyou Chen Tianbao Yang Lijun Zhang
Learning unnormalized statistical models (e.g., energy-based models) is
computationally challenging due to the complexity of handling the partition
function. To eschew this complexity, noise-contrastive estimation~(NCE) has
been proposed by formulating the objective as the logistic loss of the real
data and the artificial noise. However, as found in previous works, NCE may
perform poorly in many tasks due to its flat loss landscape and slow
convergence. In this paper, we study it a direct approach for optimizing the
negative log-likelihood of unnormalized models from the perspective of
compositional optimization. To tackle the partition function, a noise
distribution is introduced such that the log partition function can be written
as a compositional function whose inner function can be estimated with
stochastic samples. Hence, the objective can be optimized by stochastic
compositional optimization algorithms. Despite being a simple method, we
demonstrate that it is more favorable than NCE by (1) establishing a fast
convergence rate and quantifying its dependence on the noise distribution
through the variance of stochastic estimators; (2) developing better results
for one-dimensional Gaussian mean estimation by showing our objective has a
much favorable loss landscape and hence our method enjoys faster convergence;
(3) demonstrating better performance on multiple applications, including
density estimation, out-of-distribution detection, and real image generation.