Reviewing Evolution of Learning Functions and Semantic Information
  Measures for Understanding Deep Learning
Chenguang Lu
A new trend in deep learning, represented by Mutual Information Neural
Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is
emerging. In this trend, similarity functions and Estimated Mutual Information
(EMI) are used as learning and objective functions. Coincidentally, EMI is
essentially the same as Semantic Mutual Information (SeMI) proposed by the
author 30 years ago. This paper first reviews the evolutionary histories of
semantic information measures and learning functions. Then, it briefly
introduces the author's semantic information G theory with the rate-fidelity
function R(G) (G denotes SeMI, and R(G) extends R(D)) and its applications to
multi-label learning, the maximum Mutual Information (MI) classification, and
mixture models. Then it discusses how we should understand the relationship
between SeMI and Shan-non's MI, two generalized entropies (fuzzy entropy and
coverage entropy), Autoencoders, Gibbs distributions, and partition functions
from the perspective of the R(G) function or the G theory. An important
conclusion is that mixture models and Restricted Boltzmann Machines converge
because SeMI is maximized, and Shannon's MI is minimized, making information
efficiency G/R close to 1. A potential opportunity is to simplify deep learning
by using Gaussian channel mixture models for pre-training deep neural networks'
latent layers without considering gradients. It also discusses how the SeMI
measure is used as the reward function (reflecting purposiveness) for
reinforcement learning. The G theory helps interpret deep learning but is far
from enough. Combining semantic information theory and deep learning will
accelerate their development.