Max-Sliced Mutual Information
Dor Tsur Ziv Goldfeld Kristjan Greenewald
Quantifying the dependence between high-dimensional random variables is
central to statistical learning and inference. Two classical methods are
canonical correlation analysis (CCA), which identifies maximally correlated
projected versions of the original variables, and Shannon's mutual information,
which is a universal dependence measure that also captures high-order
dependencies. However, CCA only accounts for linear dependence, which may be
insufficient for certain applications, while mutual information is often
infeasible to compute/estimate in high dimensions. This work proposes a middle
ground in the form of a scalable information-theoretic generalization of CCA,
termed max-sliced mutual information (mSMI). mSMI equals the maximal mutual
information between low-dimensional projections of the high-dimensional
variables, which reduces back to CCA in the Gaussian case. It enjoys the best
of both worlds: capturing intricate dependencies in the data while being
amenable to fast computation and scalable estimation from samples. We show that
mSMI retains favorable structural properties of Shannon's mutual information,
like variational forms and identification of independence. We then study
statistical estimation of mSMI, propose an efficiently computable neural
estimator, and couple it with formal non-asymptotic error bounds. We present
experiments that demonstrate the utility of mSMI for several tasks,
encompassing independence testing, multi-view representation learning,
algorithmic fairness, and generative modeling. We observe that mSMI
consistently outperforms competing methods with little-to-no computational
overhead.