Energy-Based Models with Applications to Speech and Language Processing
Zhijian Ou
Energy-Based Models (EBMs) are an important class of probabilistic models,
also known as random fields and undirected graphical models. EBMs are
un-normalized and thus radically different from other popular self-normalized
probabilistic models such as hidden Markov models (HMMs), autoregressive
models, generative adversarial nets (GANs) and variational auto-encoders
(VAEs). Over the past years, EBMs have attracted increasing interest not only
from the core machine learning community, but also from application domains
such as speech, vision, natural language processing (NLP) and so on, due to
significant theoretical and algorithmic progress. The sequential nature of
speech and language also presents special challenges and needs a different
treatment from processing fix-dimensional data (e.g., images). Therefore, the
purpose of this monograph is to present a systematic introduction to
energy-based models, including both algorithmic progress and applications in
speech and language processing. First, the basics of EBMs are introduced,
including classic models, recent models parameterized by neural networks,
sampling methods, and various learning methods from the classic learning
algorithms to the most advanced ones. Then, the application of EBMs in three
different scenarios is presented, i.e., for modeling marginal, conditional and
joint distributions, respectively. 1) EBMs for sequential data with
applications in language modeling, where the main focus is on the marginal
distribution of a sequence itself; 2) EBMs for modeling conditional
distributions of target sequences given observation sequences, with
applications in speech recognition, sequence labeling and text generation; 3)
EBMs for modeling joint distributions of both sequences of observations and
targets, and their applications in semi-supervised learning and calibrated
natural language understanding.