With the recently rapid development in deep learning, deep neural networks have been widely adopted in many real-life applications. However, deep neural networks are also known to have very little control over its uncertainty for test examples, which potentially causes very harmful and annoying consequences in practical scenarios. In this paper, we are particularly interested in designing a higher-order uncertainty metric for deep neural networks and investigate its performance on the out-of-distribution detection task proposed by~\cite{hendrycks2016baseline}. Our method first assumes there exists a underlying higher-order distribution $\mathcal{P}(z)$ , which generated label-wise distribution $\mathcal{P}(y)$ over classes on the K-dimension simplex, and then approximate such higher-order distribution via parameterized posterior function $p_{\theta}(z|x)$ under variational inference framework, finally we use the entropy of learned posterior distribution $p_{\theta}(z|x)$ as uncertainty measure to detect out-of-distribution examples. However , we identify the overwhelming over-concentration issue in such a framework, which greatly hinders the detection performance. Therefore , we further design a log-smoothing function to alleviate such issue to greatly increase the robustness of the proposed entropy-based uncertainty measure. Through comprehensive experiments on various datasets and architectures, our proposed variational Dirichlet framework with entropy-based uncertainty measure is consistently observed to yield significant improvements over many baseline systems. Recently, deep neural networks BID18 have surged and replaced the traditional machine learning algorithms to demonstrate its potentials in many real-life applications like speech recognition BID10 , image classification BID11 , and machine translation BID34 BID32 , reading comprehension BID27 , etc. However, unlike the traditional machine learning algorithms like Gaussian Process, Logistic Regression, etc, deep neural networks are very limited in their capability to measure their uncertainty over the unseen test cases and tend to produce over-confident predictions. Such overconfidence issue BID1 ) is known to be harmful or offensive in real-life applications. Even worse, such models are prone to adversarial attacks and raise concerns in AI safety BID9 BID23 . Therefore, it is very essential to design a robust and accurate uncertainty metric in deep neural networks in order to better deploy them into real-world applications. Recently, An out-of-distribution detection task has been proposed in BID12 as a benchmark to promote the uncertainty research in the deep learning community. In the baseline approach, a simple method using the highest softmax score is adopted as the indicator for the model's confidence to distinguish in-from out-ofdistribution data. Later on, many follow-up algorithms BID21 BID19 BID30 BID4 have been proposed to achieve better performance on this benchmark. In ODIN BID21 , the authors follow the idea of temperature scaling and input perturbation BID25 to widen the distance between in-and out-of-distribution examples. Later on, adversarial training BID19 ) is introduced to explicitly introduce boundary examples as negative training data to help increase the model's robustness. In BID4 , the authors proposed to directly output a real value between [0, 1] as the confidence measure. The most recent paper BID30 leverages the semantic dense representation into the target labels to better separate the label space and uses the cosine similarity score as the confidence measure.These methods though achieve significant results on out-of-distribution detection tasks, they conflate different levels of uncertainty as pointed in BID22 . For example, when presented with two pictures, one is faked by mixing dog, cat and horse pictures, the other is a real but unseen dog, the model might output same belief as {cat:34%, dog:33%, horse:33%}. Under such scenario, the existing measures like maximum probability or label-level entropy BID21 BID30 BID12 will misclassify both images as from out-of-distribution because they are unable to separate the two uncertainty sources: whether the uncertainty is due to the data noise (class overlap) or whether the data is far from the manifold of training data. More specifically, they fail to distinguish between the lower-order (aleatoric) uncertainty BID5 , and higherorder (episdemic) uncertainty BID5 , which leads to their inferior performances in detecting out-domain examples. In order to resolve the issues presented by lower-order uncertainty measures, we are motivated to design an effective higher-order uncertainty measure for out-of-distribution detection. Inspired by Subjective Logic BID14 BID37 BID29 , we first view the label-wise distribution P(y ) as a K-dimensional variable z generated from a higher-order distribution P(z ) over the simplex S k , and then study the higher-order uncertainty by investigating the statistical properties of such underlying higher-order distribution. Under a Bayesian framework with data pair D = (x, y), we propose to use variational inference to approximate such "true" latent distribution P(z) = p(z|y ) by a parameterized Dirichlet posterior p θ (z|x), which is approximated by a deep neural network. Finally, we compute the entropy of the approximated posterior for outof-distribution detection. However, we have observed an overwhelming over-concentration problem in our experiments, which is caused by over-confidence problem of the deep neural network to greatly hinder the detection accuracy. Therefore, we further propose to smooth the Dirichlet distribution by a calibration algorithm. Combined with the input perturbation method BID21 BID16 , our proposed variational Dirichlet framework can greatly widen the distance between in-and out-of-distribution data to achieve significant results on various datasets and architectures.The contributions of this paper are described as follows:• We propose a variational Dirichlet algorithm for deep neural network classification problem and define a higher-order uncertainty measure.• We identify the over-concentration issue in our Dirichlet framework and propose a smoothing method to alleviate such problem. In this paper, we aim at finding an effective way for deep neural networks to express their uncertainty over their output distribution. Our variational Dirichlet framework is empirically demonstrated to yield better results, but its detection accuracy on a more challenging setup like CIFAR100 is still very compromised. We conjecture that better prior Dirichlet distribution or smoothing function could help further improve the performance. In the future work, we plan to apply our method to broader applications like natural language processing tasks or speech recognition tasks.• LSUN (out-of-distribution): The Large-scale Scene UNderstanding dataset (LSUN) BID38 has a test set consisting of 10,000 images from 10 different scene classes, such as bedroom, church, kitchen, and tower. We downsample LSUN's original image and create 32 × 32 images as an out-of-distribution dataset.• iSUN (out-of-distribution): The iSUN dataset BID35 ) is a subset of the SUN dataset, containing 8,925 images. All images are downsampled to 32 × 32 pixels.
A new framework based variational inference for out-of-distribution detection Describes a probabilistic approach to quantifying uncertainty in DNN classification tasks that outperforms other SOTA methods in the task of out-of-distribution detection. A new framework for out-of-distribution detection, based on variaitonal inference and a prior Dirichlet distribution, that reports state of the art results on several datasets. An out-of distribution detection via a new method to approximate the confidence distribution of classification probability using variational inference of Dirichlet distribution.