Information-theoretic Analysis of Test Data Sensitivity in Uncertainty
Futoshi Futami Tomoharu Iwata
Bayesian inference is often utilized for uncertainty quantification tasks. A
recent analysis by Xu and Raginsky 2022 rigorously decomposed the predictive
uncertainty in Bayesian inference into two uncertainties, called aleatoric and
epistemic uncertainties, which represent the inherent randomness in the
data-generating process and the variability due to insufficient data,
respectively. They analyzed those uncertainties in an information-theoretic
way, assuming that the model is well-specified and treating the model's
parameters as latent variables. However, the existing information-theoretic
analysis of uncertainty cannot explain the widely believed property of
uncertainty, known as the sensitivity between the test and training data. It
implies that when test data are similar to training data in some sense, the
epistemic uncertainty should become small. In this work, we study such
uncertainty sensitivity using our novel decomposition method for the predictive
uncertainty. Our analysis successfully defines such sensitivity using
information-theoretic quantities. Furthermore, we extend the existing analysis
of Bayesian meta-learning and show the novel sensitivities among tasks for the
first time.