How Does Information Bottleneck Help Deep Learning?
Kenji Kawaguchi Zhun Deng Xu Ji Jiaoyang Huang
Numerous deep learning algorithms have been inspired by and understood via
the notion of information bottleneck, where unnecessary information is (often
implicitly) minimized while task-relevant information is maximized. However, a
rigorous argument for justifying why it is desirable to control information
bottlenecks has been elusive. In this paper, we provide the first rigorous
learning theory for justifying the benefit of information bottleneck in deep
learning by mathematically relating information bottleneck to generalization
errors. Our theory proves that controlling information bottleneck is one way to
control generalization errors in deep learning, although it is not the only or
necessary way. We investigate the merit of our new mathematical findings with
experiments across a range of architectures and learning settings. In many
cases, generalization errors are shown to correlate with the degree of
information bottleneck: i.e., the amount of the unnecessary information at
hidden layers. This paper provides a theoretical foundation for current and
future methods through the lens of information bottleneck. Our new
generalization bounds scale with the degree of information bottleneck, unlike
the previous bounds that scale with the number of parameters, VC dimension,
Rademacher complexity, stability or robustness. Our code is publicly available
at: https://github.com/xu-ji/information-bottleneck