Differentiable Information Bottleneck for Deterministic Multi-view
  Clustering
Xiaoqiang Yan Zhixiang Jin Fengshou Han Yangdong Ye
In recent several years, the information bottleneck (IB) principle provides
an information-theoretic framework for deep multi-view clustering (MVC) by
compressing multi-view observations while preserving the relevant information
of multiple views. Although existing IB-based deep MVC methods have achieved
huge success, they rely on variational approximation and distribution
assumption to estimate the lower bound of mutual information, which is a
notoriously hard and impractical problem in high-dimensional multi-view spaces.
In this work, we propose a new differentiable information bottleneck (DIB)
method, which provides a deterministic and analytical MVC solution by fitting
the mutual information without the necessity of variational approximation.
Specifically, we first propose to directly fit the mutual information of
high-dimensional spaces by leveraging normalized kernel Gram matrix, which does
not require any auxiliary neural estimator to estimate the lower bound of
mutual information. Then, based on the new mutual information measurement, a
deterministic multi-view neural network with analytical gradients is explicitly
trained to parameterize IB principle, which derives a deterministic compression
of input variables from different views. Finally, a triplet consistency
discovery mechanism is devised, which is capable of mining the feature
consistency, cluster consistency and joint consistency based on the
deterministic and compact representations. Extensive experimental results show
the superiority of our DIB method on 6 benchmarks compared with 13
state-of-the-art baselines.