Flexible Variational Information Bottleneck: Achieving Diverse
  Compression with a Single Training
Sota Kudo Naoaki Ono Shigehiko Kanaya Ming Huang
Information Bottleneck (IB) is a widely used framework that enables the
extraction of information related to a target random variable from a source
random variable. In the objective function, IB controls the trade-off between
data compression and predictiveness through the Lagrange multiplier $\beta$.
Traditionally, to find the trade-off to be learned, IB requires a search for
$\beta$ through multiple training cycles, which is computationally expensive.
In this study, we introduce Flexible Variational Information Bottleneck (FVIB),
an innovative framework for classification task that can obtain optimal models
for all values of $\beta$ with single, computationally efficient training. We
theoretically demonstrate that across all values of reasonable $\beta$, FVIB
can simultaneously maximize an approximation of the objective function for
Variational Information Bottleneck (VIB), the conventional IB method. Then we
empirically show that FVIB can learn the VIB objective as effectively as VIB.
Furthermore, in terms of calibration performance, FVIB outperforms other IB and
calibration methods by enabling continuous optimization of $\beta$. Our codes
are available at https://github.com/sotakudo/fvib.