Justices for Information Bottleneck Theory
Faxian Cao Yongqiang Cheng Adil Mehmood Khan Zhijing Yang
This study comes as a timely response to mounting criticism of the
information bottleneck (IB) theory, injecting fresh perspectives to rectify
misconceptions and reaffirm its validity. Firstly, we introduce an auxiliary
function to reinterpret the maximal coding rate reduction method as a special
yet local optimal case of IB theory. Through this auxiliary function, we
clarify the paradox of decreasing mutual information during the application of
ReLU activation in deep learning (DL) networks. Secondly, we challenge the
doubts about IB theory's applicability by demonstrating its capacity to explain
the absence of a compression phase with linear activation functions in hidden
layers, when viewed through the lens of the auxiliary function. Lastly, by
taking a novel theoretical stance, we provide a new way to interpret the inner
organizations of DL networks by using IB theory, aligning them with recent
experimental evidence. Thus, this paper serves as an act of justice for IB
theory, potentially reinvigorating its standing and application in DL and other
fields such as communications and biomedical research.