Tighter Bounds on the Information Bottleneck with Application to Deep
  Learning
Nir Weingarten Zohar Yakhini Moshe Butman Ran Gilad-Bachrach
Deep Neural Nets (DNNs) learn latent representations induced by their
downstream task, objective function, and other parameters. The quality of the
learned representations impacts the DNN's generalization ability and the
coherence of the emerging latent space. The Information Bottleneck (IB)
provides a hypothetically optimal framework for data modeling, yet it is often
intractable. Recent efforts combined DNNs with the IB by applying VAE-inspired
variational methods to approximate bounds on mutual information, resulting in
improved robustness to adversarial attacks. This work introduces a new and
tighter variational bound for the IB, improving performance of previous
IB-inspired DNNs. These advancements strengthen the case for the IB and its
variational approximations as a data modeling framework, and provide a simple
method to significantly enhance the adversarial robustness of classifier DNNs.