Adaptive Uncertainty Estimation via High-Dimensional Testing on Latent
  Representations
Tsai Hor Chan Kin Wai Lau Jiajun Shen Guosheng Yin Lequan Yu
Uncertainty estimation aims to evaluate the confidence of a trained deep
neural network. However, existing uncertainty estimation approaches rely on
low-dimensional distributional assumptions and thus suffer from the high
dimensionality of latent features. Existing approaches tend to focus on
uncertainty on discrete classification probabilities, which leads to poor
generalizability to uncertainty estimation for other tasks. Moreover, most of
the literature requires seeing the out-of-distribution (OOD) data in the
training for better estimation of uncertainty, which limits the uncertainty
estimation performance in practice because the OOD data are typically unseen.
To overcome these limitations, we propose a new framework using data-adaptive
high-dimensional hypothesis testing for uncertainty estimation, which leverages
the statistical properties of the feature representations. Our method directly
operates on latent representations and thus does not require retraining the
feature encoder under a modified objective. The test statistic relaxes the
feature distribution assumptions to high dimensionality, and it is more
discriminative to uncertainties in the latent representations. We demonstrate
that encoding features with Bayesian neural networks can enhance testing
performance and lead to more accurate uncertainty estimation. We further
introduce a family-wise testing procedure to determine the optimal threshold of
OOD detection, which minimizes the false discovery rate (FDR). Extensive
experiments validate the satisfactory performance of our framework on
uncertainty estimation and task-specific prediction over a variety of
competitors. The experiments on the OOD detection task also show satisfactory
performance of our method when the OOD data are unseen in the training. Codes
are available at https://github.com/HKU-MedAI/bnn_uncertainty.