The ability to detect when an input sample was not drawn from the training distribution is an important  desirable property of deep neural networks. In this paper, we show that a simple ensembling of first and second order deep feature statistics can be exploited to effectively differentiate in-distribution and out-of-distribution samples. Specifically, we observe that  the mean and standard deviation within feature maps  differs greatly between in-distribution and out-of-distribution samples. Based on this observation, we propose a simple and  efficient plug-and-play detection procedure that does not require re-training, pre-processing or changes to the model.   The proposed method outperforms the state-of-the-art by a large margin in all standard benchmarking tasks, while being much simpler to implement and execute. Notably, our method improves the true negative rate from 39.6% to 95.3% when 95% of in-distribution (CIFAR-100) are correctly detected using a DenseNet and the out-of-distribution dataset is TinyImageNet resize. The source code of our method will be made publicly available. In the past few years, deep neural networks (DNNs) BID6 have settled as the state-of-the art-techniques in many difficult tasks in a plurality of domains, such as image classification BID18 , speech recognition BID9 , and machine translation BID2 BID28 . This recent progress has been mainly due to their high accuracy and good generalization ability when dealing with realworld data. Unfortunately, DNNs are also highly confident when tested against unseen samples, even if the samples are vastly different from the ones employed during training BID12 . Moreover, several works have shown that such deep networks are easily fooled by minor perturbations to the input BID5 BID27 . Obtaining a calibrated confidence score from a deep neural network is a problem under continuous investigation BID12 ) and a major thread in artificial intelligence (AI) safety BID1 . In fact, knowing when the model is wrong or inaccurate has a direct impact in many production systems, such as self-driving cars, authentication and disease identification BID0 BID7 , to name a few. BID8 showed that despite producing significantly low classification errors, DNNs confidence scores are not faithful estimates of the true certainty. Their experiments confirmed that depth, width, weight decay, and batch normalization are the main reasons for overconfident scores. Moreover, they demonstrated that a simple and yet powerful method of temperature scaling in the softmax scores is an effective way to improve calibrate a DNNs confidence. While calibrating the classifier's output to represent a faithful likelihood from the training (in-distribution) data has effective solutions, the problem of detecting whether or not the samples are generated from a known distribution (out-of-distribution), is still an open problem BID12 .One straightforward approach to calibrate the classifier's confidence in order to detect samples whose distribution differs from the training samples distribution is to train a secondary classifier that digests both in-distribution (ID) and out-of-distribution (OOD) data so that anomalies are scored differently from ID samples, as performed in BID13 . Re-training a network, however, can be computationally intensive and may even be intractable, since the number of OOD samples is virtually infinite. Other solutions rely on training both classification and generative neural net- Table 1 : Summary comparison of the characteristics of recent related methods. Test complexity refers to the required number of passes over the network. Training data is the number of samples for which the methods were calibrated against (with all standing for the whole training set). AUROC is the area under receiver characteristic curve (detailed in Section 4). Performance shown is for DenseNet trained on CIFAR-100 and using TinyImageNet (resized) as OOD dataset. Deep neural networks trained to maximize classification performance in a given dataset are extremely adapted to said dataset. The statistics of activations throughout the network for samples from the training distribution (in-distribution) are remarkably stable. However, when a sample from a different distribution (out-of-distribution) is given to the network, its activation statistics depart greatly from those of in-distribution samples. Based on this observation, we propose a very simple yet efficient method to detect out-of-distribution samples. Our method is based on computing averages of low-order statistics at the batch normalization layers of the network, and then use them as features in a linear classifier. This procedure is much simpler and efficient than current stateof-the-art methods, and outperforms them by a large margin in the traditional ID/OOD fitting task (as proposed in previous works). We evaluated all methods in the challenging task of fitting on a single OOD dataset and testing on samples from other (unseen) datasets. In this harder scenario, our method generalizes well to unseen OOD datasets, outperforming ODIN by an even larger margin. Moreover, we show some preliminary results that even in the extreme case where no OOD samples are used for the training (unsupervised) we get reasonable performance. tuning is carried out on each pair of in-and out-of-distribution samples, which is the same procedure presented in BID22 and BID21 . For reproducibility, a grid search is employed considering T ∈ {1, 1000} and with 21 linearly spaced values between 0 and 0.004 plus [0.00005, 0.0005, 0.0011].
Detecting out-of-distribution samples by using low-order feature statistics without requiring any change in underlying DNN. Presents an algorithm to detect out-of-distribution samples by using the running estimate of mean and variance within BatchNorm layers to construct feature representations later fed into a linear classifier. An approach for detecting out-of-distribution samples in which the authors propose to use logistic regression over simple statistics of each batch normalization layer of CNN. The paper suggests using Z-scores for comparing ID and OOD samples to evaluate what deep nets are trying to do.