Learning and the Unknown: Surveying Steps toward Open World Recognition
---
paper_title: Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly

paper_content:

Due to the importance of zero-shot learning, i.e. classifying images where there is a lack of labeled training data, the number of proposed approaches has recently increased steadily. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits of publicly available datasets used for this task. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g. pre-training on zero-shot test classes. Moreover, we propose a new zero-shot learning dataset, the Animals with Attributes 2 (AWA2) dataset which we make publicly available both in terms of image features and the images themselves. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss in detail the limitations of the current status of the area which can be taken as a basis for advancing it.
---
paper_title: Incremental Open Set Intrusion Recognition Using Extreme Value Machine

paper_content:

Typically, most network intrusion detection systems use supervised learning techniques to identify network anomalies. A problem exists when identifying the unknowns and automatically updating a classifier with new query classes. This is defined as an open set incremental learning problem and we propose to extend a recently introduced method, the Extreme Value Machine (EVM) to address the issue of identifying new classes during query time. The EVM is derived from the statistical extreme value theory and is the first classifier that can perform kernel-free, nonlinear, variable bandwidth outlier detection combined with incremental learning. In this paper, we utilize the EVM for intrusion detection and measure the open set recognition performance of identifying known and unknown classes. Additionally, we evaluate the performance on the KDDCUP’99 dataset and compare the results with the state-ofthe- art Weibull-SVM (W-SVM). Our findings demonstrate that the EVM mirrors the performance of the W-SVM classifier, while it supports incremental learning.
---
paper_title: Open Set Fingerprint Spoof Detection Across Novel Fabrication Materials

paper_content:

A fingerprint spoof detector is a pattern classifier that is used to distinguish a live finger from a fake (spoof) one in the context of an automated fingerprint recognition system. Most spoof detectors are learning-based and rely on a set of training images. Consequently, the performance of any such spoof detector significantly degrades when encountering spoofs fabricated using novel materials not found in the training set. In real-world applications, the problem of fingerprint spoof detection must be treated as an open set recognition problem where incomplete knowledge of the fabrication materials used to generate spoofs is present at training time, and novel materials may be encountered during system deployment. To mitigate the security risk posed by novel spoofs, this paper introduces: 1) the use of the Weibull-calibrated SVM (W-SVM), which is relatively robust for open set recognition, as a novel-material detector and a spoof detector and 2) a scheme for the automatic adaptation of the W-SVM-based spoof detector to new spoof materials that leverages interoperability across classifiers. Experiments conducted on new partitions of the LivDet 2011 database designed for open set evaluation suggest: 1) a 97% increase in the error rate of the existing spoof detectors when tested using new spoof materials and 2) up to 44% improvement in spoof detection performance across spoof materials when the proposed adaptive approach is used.
---
paper_title: On optimum recognition error and reject tradeoff

paper_content:

The performance of a pattern recognition system is characterized by its error and reject tradeoff. This paper describes an optimum rejection rule and presents a general relation between the error and reject probabilities and some simple properties of the tradeoff in the optimum recognition system. The error rate can be directly evaluated from the reject function. Some practical implications of the results are discussed. Examples in normal distributions and uniform distributions are given.
---
paper_title: On the feasibility of classification-based product package authentication

paper_content:

Depending on the product category the authenticity of a consumer good concerns economic, social and/or environmental issues. Counterfeited drugs are a threat to patient safety and cause significant economic losses. Different from physical-marking based approaches this work investigates authentication of drugs based on intrinsic texture features of the packaging material. Therefore, it is assumed that the packaging material of a certain drug shows constant but discriminative textural features which enable authentication, i.e. to prove if the packaging material is genuine or not. This objective requires considering a binary classification problem with an open set of negative classes, i.e. unknown and unseen counterfeits. In order to investigate the feasibility a novel drug packaging texture databases was acquired. The experimental evaluation of two basic requirements in texture classification serves as an evidence on the basic feasibility.
---
paper_title: Novelty detection: a review - part 1: statistical approaches

paper_content:

Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. Novelty detection is one of the fundamental requirements of a good classification or identification system since sometimes the test data contains information about objects that were not known at the time of training the model. In this paper we provide state-of-the-art review in the area of novelty detection based on statistical approaches. The second part paper details novelty detection using neural networks. As discussed, there are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics.
---
paper_title: Finding the Unknown: Novelty Detection with Extreme Value Signatures of Deep Neural Activations

paper_content:

Achieving or even surpassing human-level accuracy became recently possible in a variety of application scenarios due to the rise of convolutional neural networks (CNNs) trained from large datasets. However, solving supervised visual recognition tasks by discriminating among known categories is only one side of the coin. In contrast to this, novelty detection is still an unsolved task where instances of yet unknown categories need to be identified. Therefore, we propose to leverage the powerful discriminative nature of CNNs to novelty detection tasks by investigating class-specific activation patterns. More precisely, we assume that a semantic category can be described by its extreme value signature, that specifies which dimensions of deep neural activations have largest values. By following this intuition, we show that already a small number of high-valued dimensions allows to separate known from unknown categories. Our approach is simple, intuitive, and can be easily put on top of CNNs trained for vanilla classification tasks. We empirically validate the benefits of our approach in terms of accuracy and speed by comparing it against established methods in a variety of novelty detection tasks derived from ImageNet. Finally, we show that visualizing extreme value signatures allows to inspect class-specific patterns learned during training which may ultimately help to better understand CNN models.
---
paper_title: Data-Fusion Techniques for Open-Set Recognition Problems

paper_content:

Most pattern classification techniques are focused on solving closed-set problems in which a classifier is trained with samples of all classes that may appear during the testing phase. In many situations, however, samples of unknown classes, i.e., whose classes did not have any example during the training stage, need to be properly handled during testing. This specific setup is referred to in the literature as open-set recognition. Open-set problems are harder as they might be ill-sampled, not sampled at all, or even undefined. Differently from existing literature, here we aim at solving open-set recognition problems combining different classifiers and features while, at the same time, taking care of unknown classes. Researchers have greatly benefited from combining different methods in order to achieve more robust and reliable classifiers in daring recognition conditions, but those solutions have often focused on closed-set setups. In this paper, we propose the integration of a newly designed open-set graph-based optimum-path forest (OSOPF) classifier with genetic programming (GP) and majority voting fusion techniques. While OSOPF takes care of learning decision boundaries more resilient to unknown classes and outliers, GP combines different problem features to discover appropriate similarity functions and allows a more robust classification through early fusion. Finally, the majority-voting approach combines different classification evidence from different classifier outcomes and features through late-fusion techniques. Performed experiments show the proposed data-fusion approaches yield effective results for open-set recognition problems, significantly outperforming existing counterparts in the literature and paving the way for investigations in this field.
---
paper_title: Towards Open World Recognition

paper_content:

With the of advent rich classification models and high computational power visual recognition systems have found many operational applications. Recognition in the real world poses multiple challenges that are not apparent in controlled lab environments. The datasets are dynamic and novel categories must be continuously detected and then added. At prediction time, a trained system has to deal with myriad unseen categories. Operational systems require minimal downtime, even to learn. To handle these operational issues, we present the problem of Open World Recognition and formally define it. We prove that thresholding sums of monotonically decreasing functions of distances in linearly transformed feature space can balance “open space risk” and empirical risk. Our theory extends existing algorithms for open world recognition. We present a protocol for evaluation of open world recognition systems. We present the Nearest Non-Outlier (NNO) algorithm that evolves model efficiently, adding object categories incrementally while detecting outliers and managing open space risk. We perform experiments on the ImageNet dataset with 1.2M+ images to validate the effectiveness of our method on large scale visual recognition tasks. NNO consistently yields superior results on open world recognition.
---
paper_title: Using Visual Rhythms for Detecting Video-Based Facial Spoof Attacks

paper_content:

Spoofing attacks or impersonation can be easily accomplished in a facial biometric system wherein users without access privileges attempt to authenticate themselves as valid users, in which an impostor needs only a photograph or a video with facial information of a legitimate user. Even with recent advances in biometrics, information forensics and security, vulnerability of facial biometric systems against spoofing attacks is still an open problem. Even though several methods have been proposed for photo-based spoofing attack detection, attacks performed with videos have been vastly overlooked, which hinders the use of the facial biometric systems in modern applications. In this paper, we present an algorithm for video-based spoofing attack detection through the analysis of global information which is invariant to content, since we discard video contents and analyze content-independent noise signatures present in the video related to the unique acquisition processes. Our approach takes advantage of noise signatures generated by the recaptured video to distinguish between fake and valid access videos. For that, we use the Fourier spectrum followed by the computation of video visual rhythms and the extraction of different characterization methods. For evaluation, we consider the novel unicamp video-attack database, which comprises 17 076 videos composed of real access and spoofing attack videos. In addition, we evaluate the proposed method using the replay-attack database, which contains photo-based and video-based face spoofing attacks.
---
paper_title: A bounded neural network for open set recognition

paper_content:

Open set recognition is, more than an interesting research subject, a component of various machine learning applications which is sometimes neglected: it is not unusual the existence of learning systems developed on the top of closed-set assumptions, ignoring the error risk involved in a prediction. This risk is strictly related to the location in feature space where the prediction has to be made, compared to the location of the training data: the more distant the training observations are, less is known, higher is the risk. Proper handling of this risk can be necessary in various situation where classification and its variants are employed. This paper presents an approach to open set recognition based on an elaborate distance-like computation provided by a weightless neural network model. The results obtained in the proposed test scenarios are quite interesting, placing the proposed method among the current best ones.
---
paper_title: Connecting the dots: Toward accountable machine-learning printer attribution methods

paper_content:

Abstract Digital forensics is rapidly evolving as a direct consequence of the adoption of machine-learning methods allied with ever-growing amounts of data. Despite the fact that these methods yield more consistent and accurate results, they may face adoption hindrances in practice if their produced results are absent in a human-interpretable form. In this paper, we exemplify how human-interpretable (a.k.a., accountable) extensions can enhance existing algorithms to aid human experts, by introducing a new method for the source printer attribution problem. We leverage the recently proposed Convolutional Texture Gradient Filter (CTGF) algorithm’s ability to capture local printing imperfections to introduce a new method that maps and highlights important attribution features directly onto the investigated printed document. Supported by Random Forest classifiers, we isolate and rank features that are pivotal for differentiating a printer from others, and back-project those features onto the investigated document, giving analysts further evidence about the attribution process.
---
paper_title: Novelty detection and multi-class classification in power distribution voltage waveforms

paper_content:

Accurate classification of events in waveforms from electrical distribution networks.Novelty detection: dynamic identification of new classes of events.SVDD using negative examples and maximal margin separation: better generalization.Experiments using real data: significant improvements in classification accuracy.Direct application as part of tools to assist mitigation processes in power utilities. The automatic analysis of electrical waveforms is a recurring subject in the power system sector worldwide. In this sense, the idea of this paper is to present an original approach for automatic classification of voltage waveforms in electrical distribution networks. It includes both the classification of the waveforms in multiple known classes, and the detection of new waveforms (novelties) that are not available during the training stage. The classification method, based on the Support Vector Data Description (SVDD), has a suitable formulation for this task, because it is capable of fitting a model on a relatively small set of examples, which may also include negative examples (patterns from other known classes or even novelties), with maximal margin separation. The results obtained on both simulated and real world data demonstrate the ability of the method to identify novelties and to classify known examples correctly. The method finds application in the mitigation process of emergencies normally performed by power utilities' maintenance and protection engineers, which requires fast and accurate event cause identification.
---
paper_title: Learning Person-Specific Representations From Faces in the Wild

paper_content:

Humans are natural face recognition experts, far out-performing current automated face recognition algorithms, especially in naturalistic, “in the wild” settings. However, a striking feature of human face recognition is that we are dramatically better at recognizing highly familiar faces, presumably because we can leverage large amounts of past experience with the appearance of an individual to aid future recognition. Meanwhile, the analogous situation in automated face recognition, where a large number of training examples of an individual are available, has been largely underexplored, in spite of the increasing relevance of this setting in the age of social media. Inspired by these observations, we propose to explicitly learn enhanced face representations on a per-individual basis, and we present two methods enabling this approach. By learning and operating within person-specific representations, we are able to significantly outperform the previous state-of-the-art on PubFig83, a challenging benchmark for familiar face recognition in the wild, using a novel method for learning representations in deep visual hierarchies. We suggest that such person-specific representations aid recognition by introducing an intermediate form of regularization to the problem.
---
paper_title: Open set intrusion recognition for fine-grained attack categorization

paper_content:

Confidently distinguishing a malicious intrusion over a network is an important challenge. Most intrusion detection system evaluations have been performed in a closed set protocol in which only classes seen during training are considered during classification. Thus far, there has been no realistic application in which novel types of behaviors unseen at training - unknown classes as it were - must be recognized for manual categorization. This paper comparatively evaluates malware classification using both closed set and open set protocols for intrusion recognition on the KDDCUP'99 dataset. In contrast to much of the previous work, we employ a fine-grained recognition protocol, in which the dataset is loosely open set - i.e., recognizing individual intrusion types - e.g., “sendmail”, “snmp_guess”, …, etc., rather than more general attack categories (e.g., “DoS”,“Probe”,“R2L”,“U2R”,“Normal”). We also employ two different classifier types - Gaussian RBF kernel SVMs, which are not theoretically guaranteed to bound open space risk, and W-SVMs, which are theoretically guaranteed to bound open space risk. We find that the W-SVM offers superior performance under the open set regime, particularly as the cost of misclassifying unknown classes at query time (i.e., classes not present in the training set) increases. Results of performance tradeoff with respect to cost of unknown as well as discussion of the ramifications of these findings in an operational setting are presented.
---
paper_title: Convolutional Neural Network approaches to granite tiles classification

paper_content:

Abstract The quality control process in stone industry is a challenging problem to deal with nowadays. Due to the similar visual appearance of different rocks with the same mineralogical content, economical losses can happen in industry if clients cannot recognize properly the rocks delivered as the ones initially purchased. In this paper, we go toward the automation of rock-quality assessment in different image resolutions by proposing the first data-driven technique applied to granite tiles classification. Our approach understands intrinsic patterns in small image patches through the use of Convolutional Neural Networks tailored for this problem. Experiments comparing the proposed approach to texture descriptors in a well-known dataset show the effectiveness of the proposed method and its suitability for applications in some uncontrolled conditions, such as classifying granite tiles under different image resolutions.
---
paper_title: iCaRL: Incremental Classifier and Representation Learning

paper_content:

A major open problem on the road to artificial intelligence is the development of incrementally learning systems that learn about more and more concepts over time from a stream of data. In this work, we introduce a new training strategy, iCaRL, that allows learning in such a class-incremental way: only the training data for a small number of classes has to be present at the same time and new classes can be added progressively. iCaRL learns strong classifiers and a data representation simultaneously. This distinguishes it from earlier works that were fundamentally limited to fixed data representations and therefore incompatible with deep learning architectures. We show by experiments on CIFAR-100 and ImageNet ILSVRC 2012 data that iCaRL can learn many classes incrementally over a long period of time where other strategies quickly fail.
---
paper_title: Place categorization and semantic mapping on a mobile robot

paper_content:

In this paper we focus on the challenging problem of place categorization and semantic mapping on a robot without environment-specific training. Motivated by their ongoing success in various visual recognition tasks, we build our system upon a state-of-the-art convolutional network. We overcome its closed-set limitations by complementing the network with a series of one-vs-all classifiers that can learn to recognize new semantic classes online. Prior domain knowledge is incorporated by embedding the classification system into a Bayesian filter framework that also ensures temporal coherence. We evaluate the classification accuracy of the system on a robot that maps a variety of places on our campus in real-time. We show how semantic information can boost robotic object detection performance and how the semantic map can be used to modulate the robot's behaviour during navigation tasks. The system is made available to the community as a ROS module.
---
paper_title: Classification Under Streaming Emerging New Classes: A Solution Using Completely-Random Trees

paper_content:

This paper investigates an important problem in stream mining, i.e., classification under streaming emerging new classes or SENC . The SENC problem can be decomposed into three subproblems: detecting emerging new classes, classifying known classes, and updating models to integrate each new class as part of known classes. The common approach is to treat it as a classification problem and solve it using either a supervised learner or a semi-supervised learner. We propose an alternative approach by using unsupervised learning as the basis to solve this problem. The proposed method employs completely-random trees which have been shown to work well in unsupervised learning and supervised learning independently in the literature. The completely-random trees are used as a single common core to solve all three subproblems: unsupervised learning, supervised learning, and model update on data streams. We show that the proposed unsupervised-learning-focused method often achieves significantly better outcomes than existing classification-focused methods.
---
paper_title: Probability Models for Open Set Recognition

paper_content:

Real-world tasks in computer vision often touch upon open set recognition: multi-class recognition with incomplete knowledge of the world and many unknown inputs. Recent work on this problem has proposed a model incorporating an open space risk term to account for the space beyond the reasonable support of known classes. This paper extends the general idea of open space risk limiting classification to accommodate non-linear classifiers in a multi-class setting. We introduce a new open set recognition model called compact abating probability (CAP), where the probability of class membership decreases in value (abates) as points move from known data toward open space. We show that CAP models improve open set recognition for multiple algorithms. Leveraging the CAP formulation, we go on to describe the novel Weibull-calibrated SVM (W-SVM) algorithm, which combines the useful properties of statistical extreme value theory for score calibration with one-class and binary support vector machines. Our experiments show that the W-SVM is significantly better for open set object detection and OCR problems when compared to the state-of-the-art for the same tasks.
---
paper_title: Modular ensembles for one-class classification based on density analysis

paper_content:

One-Class Classification (OCC) is an important machine learning task. It studies a special classification problem that training samples from only one class, named target class, are available or reliable. Recently, various OCC algorithms have been proposed, however many of them do not adequately deal with multi-modality, multi-density, the noise and arbitrarily shaped distributions of the target class. In this paper, we propose a novel Density Based Modular Ensemble One-class Classifier (DBM-EOC) algorithm which is motivated by density analysis, divide-and-conquer method and ensemble learning. DBM-EOC first performs density analysis on training samples to obtain a minimal spanning tree using density characteristics of the target class. On this basis, DBM-EOC automatically identifies clusters, multi-density distributions and the noise in training samples using extreme value analysis. Then target samples are categorized into several groups called Local Dense Subset (LDS). Samples in each LDS are close to each other and their local densities are similar. A simple base OCC model e.g. the Gaussian estimator is built for each LDS afterwards. Finally all the base classifiers are modularly aggregated to construct the DBM-EOC model. We experimentally evaluate DBM-EOC with 6 state-of-art OCC algorithms on 5 synthetic datasets, 18 UCI benchmark datasets and the MNIST dataset. The results show that DBM-EOC outperforms other competitors in majority cases especially when the datasets are multi-modality, multi-density or noisy. We propose a modular ensemble OCC algorithm DBM-EOC based on density analysis.We analyze peculiarities of the target class which are crucial for OCC.DBM-EOC obtains a tree structure of the target class considering density.DBM-EOC can automatically detect clusters and remove noise samples.DBM-EOC solves OCC problems with the divide-and-conquer method.
---
paper_title: Challenges in detecting UAS with radar

paper_content:

The recent proliferation of drones has contributed to the emergence of new threats in security applications. Because of their great agility and small size, UAS can be used for numerous missions and are very challenging to detect. Radar technology with its all-weather capability can play an important role in detecting UAS-based threats and in protecting critical assets. However, to be successful, radars have to quickly scan large volumes with great sensitivity, eliminate nuisance alarms from birds and discriminate UAS from ground targets. Radar parameters, antenna scan techniques and target classification requirements for UAS detection are analyzed. A radar implementation is discussed and preliminary results presented. Overall, an X-band radar with electronic scanning capability can contribute to a reliable and affordable solution for detecting UAS-based threats.
---
paper_title: Local Novelty Detection in Multi-class Recognition Problems

paper_content:

In this paper, we propose using local learning for multiclass novelty detection, a framework that we call local novelty detection. Estimating the novelty of a new sample is an extremely challenging task due to the large variability of known object categories. The features used to judge on the novelty are often very specific for the object in the image and therefore we argue that individual novelty models for each test sample are important. Similar to human experts, it seems intuitive to first look for the most related images thus filtering out unrelated data. Afterwards, the system focuses on discovering similarities and differences to those images only. Therefore, we claim that it is beneficial to solely consider training images most similar to a test sample when deciding about its novelty. Following the principle of local learning, for each test sample a local novelty detection model is learned and evaluated. Our local novelty score turns out to be a valuable indicator for deciding whether the sample belongs to a known category from the training set or to a new, unseen one. With our local novelty detection approach, we achieve state-of-the-art performance in multi-class novelty detection on two popular visual object recognition datasets, Caltech-256 and Image Net. We further show that our framework: (i) can be successfully applied to unknown face detection using the Labeled-Faces-in-the-Wild dataset and (ii) outperforms recent work on attribute-based unfamiliar class detection in fine-grained recognition of bird species on the challenging CUB-200-2011 dataset.
---
paper_title: Audio Event Recognition in the Smart Home

paper_content:

After giving a brief overview of the relevance and value of deploying automatic audio event recognition (AER) in the smart home market, this chapter reviews three aspects of the productization of AER which are important to consider when developing pathways to impact between fundamental research and “real-world” applicative outlets. In the first section, it is shown that applications introduce a variety of practical constraints which elicit new research topics in the field: clarifying the definition of sound events, thus suggesting interest for the explicit modeling of temporal patterns and interruption; running and evaluating AER in 24/7 sound detection setups, which suggests to recast the problem as open-set recognition; and running AER applications on consumer devices with limited audio quality and computational power, thus triggering interest for scalability and robustness. The second section explores the definition of user experience for AER. After reporting field observations about the ways in which system errors affect user experience, it is proposed to introduce opinion scoring into AER evaluation methodology. Then, the link between standard AER performance metrics and subjective user experience metrics is being explored, and attention is being drawn to the fact that F-score metrics actually mash up the objective evaluation of acoustic discrimination with the subjective choice of an application-dependent operation point. Solutions to the separation of discrimination and calibration in system evaluation are introduced, thus allowing the more explicit separation of acoustic modeling optimization from that of application-dependent user experience. Finally, the last section analyses the ethical and legal issues involved in deploying AER systems which are “listening” at all times into the users’ private space. A review of the key notions underpinning European data and privacy protection laws, questioning if and when these apply to audio data, suggests a set of guidelines which summarize into empowering users to consent by fully informing them about the use of their data, as well as taking reasonable information security measures to protect users’ personal data.
---
paper_title: Authorship Attribution for Social Media Forensics

paper_content:

The veil of anonymity provided by smartphones with pre-paid SIM cards, public Wi-Fi hotspots, and distributed networks like Tor has drastically complicated the task of identifying users of social media during forensic investigations. In some cases, the text of a single posted message will be the only clue to an author’s identity. How can we accurately predict who that author might be when the message may never exceed 140 characters on a service like Twitter? For the past 50 years, linguists, computer scientists, and scholars of the humanities have been jointly developing automated methods to identify authors based on the style of their writing. All authors possess peculiarities of habit that influence the form and content of their written works. These characteristics can often be quantified and measured using machine learning algorithms. In this paper, we provide a comprehensive review of the methods of authorship attribution that can be applied to the problem of social media forensics. Furthermore, we examine emerging supervised learning-based methods that are effective for small sample sizes, and provide step-by-step explanations for several scalable approaches as instructional case studies for newcomers to the field. We argue that there is a significant need in forensics for new authorship attribution algorithms that can exploit context, can process multi-modal data, and are tolerant to incomplete knowledge of the space of all possible authors at training time.
---
paper_title: An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild

paper_content:

We investigate the problem of generalized zero-shot learning (GZSL). GZSL relaxes the unrealistic assumption in conventional zero-shot learning (ZSL) that test data belong only to unseen novel classes. In GZSL, test data might also come from seen classes and the labeling space is the union of both types of classes. We show empirically that a straightforward application of classifiers provided by existing ZSL approaches does not perform well in the setting of GZSL. Motivated by this, we propose a surprisingly simple but effective method to adapt ZSL approaches for GZSL. The main idea is to introduce a calibration factor to calibrate the classifiers for both seen and unseen classes so as to balance two conflicting forces: recognizing data from seen classes and those from unseen ones. We develop a new performance metric called the Area Under Seen-Unseen accuracy Curve to characterize this trade-off. We demonstrate the utility of this metric by analyzing existing ZSL approaches applied to the generalized setting. Extensive empirical studies reveal strengths and weaknesses of those approaches on three well-studied benchmark datasets, including the large-scale ImageNet with more than 20,000 unseen categories. We complement our comparative studies in learning methods by further establishing an upper bound on the performance limit of GZSL. In particular, our idea is to use class-representative visual features as the idealized semantic embeddings. We show that there is a large gap between the performance of existing approaches and the performance limit, suggesting that improving the quality of class semantic embeddings is vital to improving ZSL.
---
paper_title: Open Set Domain Adaptation

paper_content:

When the training and the test data belong to different domains, the accuracy of an object classifier is significantly reduced. Therefore, several algorithms have been proposed in the last years to diminish the so called domain shift between datasets. However, all available evaluation protocols for domain adaptation describe a closed set recognition task, where both domains, namely source and target, contain exactly the same object classes. In this work, we also explore the field of domain adaptation in open sets, which is a more realistic scenario where only a few categories of interest are shared between source and target data. Therefore, we propose a method that fits in both closed and open set scenarios. The approach learns a mapping from the source to the target domain by jointly solving an assignment problem that labels those target instances that potentially belong to the categories of interest present in the source dataset. A thorough evaluation shows that our approach outperforms the state-of-the-art.
---
paper_title: Zero-Shot Learning — The Good, the Bad and the Ugly

paper_content:

Due to the importance of zero-shot learning, the number of proposed approaches has increased steadily recently. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g. pre-training on zero-shot test classes. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss limitations of the current status of the area which can be taken as a basis for advancing it.
---
paper_title: Overcoming the challenge for text classification in the open world

paper_content:

Classification is often referred to as the task of discriminating one class from others in a given set of classes. Traditionally, classifiers work well assuming that a priori knowledge of all classes are given. Unfortunately, a presenting of unknown class during testing can lead to poor performance of even state-of-the-art classifiers due to observed classes being incorrectly identified to other classes. Recent proposed open world recognition framework provides a promising venue for tackling this challenge. While the majority of works in this relative new field is in computer vision, the rare work in Natural Language Processing shows its instability in its performance and is not based on the open world recognition framework. To tackle this problem, we represent our Nearest Centroid Class (NCC) which is incremental learning and able to detect unknown class during testing. Our model yields promising results in a document classification on text classification domains among current state-of-the-art models.
---
paper_title: Towards Open-Set Identity Preserving Face Synthesis

paper_content:

We propose a framework based on Generative Adversarial Networks to disentangle the identity and attributes of faces, such that we can conveniently recombine different identities and attributes for identity preserving face synthesis in open domains. Previous identity preserving face synthesis processes are largely confined to synthesizing faces with known identities that are already in the training dataset. To synthesize a face with identity outside the training dataset, our framework requires one input image of that subject to produce an identity vector, and any other input face image to extract an attribute vector capturing, e.g., pose, emotion, illumination, and even the background. We then recombine the identity vector and the attribute vector to synthesize a new face of the subject with the extracted attribute. Our proposed framework does not need to annotate the attributes of faces in any way. It is trained with an asymmetric loss function to better preserve the identity and stabilize the training process. It can also effectively leverage large amounts of unlabeled training face images to further improve the fidelity of the synthesized faces for subjects that are not presented in the labeled training face dataset. Our experiments demonstrate the efficacy of the proposed framework. We also present its usage in a much broader set of applications including face frontalization, face attribute morphing, and face adversarial example detection.
---
paper_title: Open Set Learning with Counterfactual Images

paper_content:

In open set recognition, a classifier must label instances of known classes while detecting instances of unknown classes not encountered during training. To detect unknown classes while still generalizing to new instances of existing classes, we introduce a dataset augmentation technique that we call counterfactual image generation. Our approach, based on generative adversarial networks, generates examples that are close to training set examples yet do not belong to any training category. By augmenting training with examples generated by this optimization, we can reformulate open set recognition as classification with one additional class, which includes the set of novel and unknown examples. Our approach outperforms existing open set recognition algorithms on a selection of image classification tasks.
---
paper_title: RO-SVM: Support Vector Machine with Reject Option for Image Categorization

paper_content:

When applying Multiple Instance Learning (MIL) for image categorization, an image is treated as a bag containing a number of instances, each representing a region inside the image. The categorization of this image is determined by the labels of these instances, which are not specified in the training data-set. Hence, these instance labels are needed to be estimated together with the classifier. To improve classification reliability, we propose in this paper a new Support Vector Machine approach by incorporating a reject option, named RO-SVM to determine the instance labels, and the rejection region during the training phase simultaneously. Our approach can also be easily extended to solve multi-class classification problems. Experimental results demonstrate that higher categorization accuracy can be achieved with our RO-SVM method, comparing to approaches that do not exclude uninformative image patches. Our method is able to produce results comparable even with few training samples.
---
paper_title: Extreme Value Analysis for Mobile Active User Authentication

paper_content:

In this paper, we propose to improve the performance of mobile Active Authentication (AA) systems in the low false alarm region using the statistical Extreme Value Theory (EVT). The problem is studied under a Bayesian framework where extremal observations that contribute to mis-verification are given more prominence. We prop~e modeling !he.tail. of the match distribution using a Generalized Pareto Distrfbution (GPD) in order to make better inferences about the extremal observations. A method based on the mean excess function is introduced for parameter estimation of the GPD. Effectiveness of the proposed framework is demonstrated using publicly available unconstrained mobile active authentication datasets. It is shown that the proposed EVT-based method can significantly enhance the performance of traditional AA systems in the low false alarm rate region.
---
paper_title: A Survey of Stealth Malware Attacks, Mitigation Measures, and Steps Toward Autonomous Open World Solutions

paper_content:

As our professional, social, and financial existences become increasingly digitized and as our government, healthcare, and military infrastructures rely more on computer technologies, they present larger and more lucrative targets for malware. Stealth malware in particular poses an increased threat because it is specifically designed to evade detection mechanisms, spreading dormant, in the wild for extended periods of time, gathering sensitive information or positioning itself for a high-impact zero-day attack. Policing the growing attack surface requires the development of efficient anti-malware solutions with improved generalization to detect novel types of malware and resolve these occurrences with as little burden on human experts as possible. In this paper, we survey malicious stealth technologies as well as existing solutions for detecting and categorizing these countermeasures autonomously. While machine learning offers promising potential for increasingly autonomous solutions with improved generalization to new malware types, both at the network level and at the host level, our findings suggest that several flawed assumptions inherent to most recognition algorithms prevent a direct mapping between the stealth malware recognition problem and a machine learning solution. The most notable of these flawed assumptions is the closed world assumption: that no sample belonging to a class outside of a static training set will appear at query time. We present a formalized adaptive open world framework for stealth malware recognition and relate it mathematically to research from other machine learning domains.
---
paper_title: Open set recognition for automatic target classification with rejection

paper_content:

Training sets for supervised classification tasks are usually limited in scope and only contain examples of a few classes. In practice, classes that were not seen in training are given labels that are always incorrect. Open set recognition (OSR) algorithms address this issue by providing classifiers with a rejection option for unknown samples. In this work, we introduce a new OSR algorithm and compare its performance to other current approaches for open set image classification.
---
paper_title: Open set source camera attribution and device linking

paper_content:

Camera attribution approaches in digital image forensics have most often been evaluated in a closed set context, whereby all devices are known during training and testing time. However, in a real investigation, we must assume that innocuous images from unknown devices will be recovered, which we would like to remove from the pool of evidence. In pattern recognition, this corresponds to what is known as the open set recognition problem. This article introduces new algorithms for open set modes of image source attribution (identifying whether or not an image was captured by a specific digital camera) and device linking (identifying whether or not a pair of images was acquired from the same digital camera without the need for physical access to the device). Both algorithms rely on a new multi-region feature generation strategy, which serves as a projection space for the class of interest and emphasizes its properties, and on decision boundary carving, a novel method that models the decision space of a trained SVM classifier by taking advantage of a few known cameras to adjust the decision boundaries to decrease false matches from unknown classes. Experiments including thousands of unconstrained images collected from the web show a significant advantage for our approaches over the most competitive prior work.
---
paper_title: Open set recognition of aircraft in aerial imagery using synthetic template models

paper_content:

Fast, accurate and robust automatic target recognition (ATR) in optical aerial imagery can provide game-changing ::: advantages to military commanders and personnel. ATR algorithms must reject non-targets with a high degree of ::: confidence in a world with an infinite number of possible input images. Furthermore, they must learn to recognize new ::: targets without requiring massive data collections. Whereas most machine learning algorithms classify data in a closed set ::: manner by mapping inputs to a fixed set of training classes, open set recognizers incorporate constraints that allow for ::: inputs to be labelled as unknown. We have adapted two template-based open set recognizers to use computer generated ::: synthetic images of military aircraft as training data, to provide a baseline for military-grade ATR: (1) a frequentist ::: approach based on probabilistic fusion of extracted image features, and (2) an open set extension to the one-class support ::: vector machine (SVM). These algorithms both use histograms of oriented gradients (HOG) as features as well as artificial ::: augmentation of both real and synthetic image chips to take advantage of minimal training data. Our results show that ::: open set recognizers trained with synthetic data and tested with real data can successfully discriminate real target inputs ::: from non-targets. However, there is still a requirement for some knowledge of the real target in order to calibrate the ::: relationship between synthetic template and target score distributions. We conclude by proposing algorithm modifications ::: that may improve the ability of synthetic data to represent real data.
---
paper_title: Analyzing the Roles of Descriptions and Actions in Open Systems

paper_content:

This paper analyzes relationships between the roles of descriptions and actions in large scale, open ended, geographically distributed, concurrent systems. Rather than attempt to deal with the complexities and ambiguities of currently implemented descriptive languages, we concentrate our analysis on what can be expressed in the underlying frameworks such as the lambda calculus and first order logic. By this means we conclude that descriptions and actions complement one another; neither being sufficient unto itself. This paper provides a basis to begin the analysis of the very subtle relationships that hold between descriptions and actions in Open Systems.
---
paper_title: Metric learning for large scale image classification: generalizing to new classes at near-zero cost

paper_content:

We are interested in large-scale image classification and especially in the setting where images corresponding to new or existing classes are continuously added to the training set. Our goal is to devise classifiers which can incorporate such images and classes on-the-fly at (near) zero cost. We cast this problem into one of learning a metric which is shared across all classes and explore k-nearest neighbor (k-NN) and nearest class mean (NCM) classifiers. We learn metrics on the ImageNet 2010 challenge data set, which contains more than 1.2M training images of 1K classes. Surprisingly, the NCM classifier compares favorably to the more flexible k-NN classifier, and has comparable performance to linear SVMs. We also study the generalization performance, among others by using the learned metric on the ImageNet-10K dataset, and we obtain competitive performance. Finally, we explore zero-shot classification, and show how the zero-shot model can be combined very effectively with small training datasets.
---
paper_title: Domain Adaptation for Visual Recognition

paper_content:

Domain adaptation is an active, emerging research area that attemptsto address the changes in data distribution across training and testingdatasets. With the availability of a multitude of image acquisition sensors,variations due to illumination, and viewpoint among others, computervision applications present a very natural test bed for evaluatingdomain adaptation methods. In this monograph, we provide a comprehensiveoverview of domain adaptation solutions for visual recognitionproblems. By starting with the problem description and illustrations,we discuss three adaptation scenarios namely, i unsupervised adaptationwhere the "source domain" training data is partially labeledand the "target domain" test data is unlabeled, ii semi-supervisedadaptation where the target domain also has partial labels, and iiimulti-domain heterogeneous adaptation which studies the previous twosettings with the source and/or target having more than one domain,and accounts for cases where the features used to represent the datain each domain are different. For all these topics we discuss existingadaptation techniques in the literature, which are motivated by theprinciples of max-margin discriminative learning, manifold learning,sparse coding, as well as low-rank representations. These techniqueshave shown improved performance on a variety of applications suchas object recognition, face recognition, activity analysis, concept classification,and person detection. We then conclude by analyzing thechallenges posed by the realm of "big visual data", in terms of thegeneralization ability of adaptation algorithms to unconstrained dataacquisition as well as issues related to their computational tractability,and draw parallels with the efforts from vision community on imagetransformation models, and invariant descriptors so as to facilitate improvedunderstanding of vision problems under uncertainty.
---
paper_title: Support Vector Machines with Embedded Reject Option

paper_content:

In this paper, the problem of implementing the reject option in support vector machines (SVMs) is addressed. We started by observing that methods proposed so far simply apply a reject threshold to the outputs of a trained SVM. We then showed that, under the framework of the structural risk minimisation principle, the rejection region must be determined during the training phase of a classifier. By applying this concept, and by following Vapnik's approach, we developed a maximum margin classifier with reject option. This led us to a SVM whose rejection region is determined during the training phase, that is, a SVM with embedded reject option. To implement such a SVM, we devised a novel formulation of the SVM training problem and developed a specific algorithm to solve it. Preliminary results on a character recognition problem show the advantages of the proposed SVM in terms of the achievable error-reject trade-off.
---
paper_title: Steps Toward Robust Artificial Intelligence

paper_content:

Recent advances in artificial intelligence are encouraging governments and corporations to deploy AI in high-stakes settings including driving cars autonomously, managing the power grid, trading on stock exchanges, and controlling autonomous weapons systems. Such applications require AI methods to be robust to both the known unknowns (those uncertain aspects of the world about which the computer can reason explicitly) and the unknown unknowns (those aspects of the world that are not captured by the system’s models). This article discusses recent progress in AI and then describes eight ideas related to robustness that are being pursued within the AI research community. While these ideas are a start, we need to devote more attention to the challenges of dealing with the known and unknown unknowns. These issues are fascinating, because they touch on the fundamental question of how finite systems can survive and thrive in a complex and dangerous world
---
paper_title: Attribute-Based Classification for Zero-Shot Visual Object Categorization

paper_content:

We study the problem of object recognition for categories for which we have no training examples, a task also called zero--data or zero-shot learning. This situation has hardly been studied in computer vision research, even though it occurs frequently; the world contains tens of thousands of different object classes, and image collections have been formed and suitably annotated for only a few of them. To tackle the problem, we introduce attribute-based classification: Objects are identified based on a high-level description that is phrased in terms of semantic attributes, such as the object's color or shape. Because the identification of each such property transcends the specific learning task at hand, the attribute classifiers can be prelearned independently, for example, from existing image data sets unrelated to the current task. Afterward, new classes can be detected based on their attribute representation, without the need for a new training phase. In this paper, we also introduce a new data set, Animals with Attributes, of over 30,000 images of 50 animal classes, annotated with 85 semantic attributes. Extensive experiments on this and two more data sets show that attribute-based classification indeed is able to categorize images without access to any training images of the target classes.
---
paper_title: Lifelong Machine Learning

paper_content:

Abstract NOTE ⁃ A New Edition of This Title is Available: Lifelong Machine Learning, Second Edition Lifelong Machine Learning (or Lifelong Learning) is an advanced machine learning paradigm that le...
---
paper_title: Unbounded cache model for online language modeling with open vocabulary

paper_content:

Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of representations while searching them efficiently. We conduct extensive experiments showing that our approach significantly improves the perplexity of pre-trained language models on new distributions, and can scale efficiently to much larger contexts than previously proposed local cache models.
---
paper_title: Multi-class Fukunaga Koontz discriminant analysis for enhanced face recognition

paper_content:

Linear subspace learning methods such as Fisher's Linear Discriminant Analysis (LDA), Unsupervised Discriminant Projection (UDP), and Locality Preserving Projections (LPP) have been widely used in face recognition applications as a tool to capture low dimensional discriminant information. However, when these methods are applied in the context of face recognition, they often encounter the small-sample-size problem. In order to overcome this problem, a separate Principal Component Analysis (PCA) step is usually adopted to reduce the dimensionality of the data. However, such a step may discard dimensions that contain important discriminative information that can aid classification performance. In this work, we propose a new idea which we named Multi-class Fukunaga Koontz Discriminant Analysis (FKDA) by incorporating the Fukunaga Koontz Transform within the optimization for maximizing class separation criteria in LDA, UDP, and LPP. In contrast to traditional LDA, UDP, and LPP, our approach can work with very high dimensional data as input, without requiring a separate dimensionality reduction step to make the scatter matrices full rank. In addition, the FKDA formulation seeks optimal projection direction vectors that are orthogonal which the existing methods cannot guarantee, and it has the capability of finding the exact solutions to the "trace ratio" objective in discriminant analysis problems while traditional methods can only deal with a relaxed and inexact "ratio trace" objective. We have shown using six face database, in the context of large scale unconstrained face recognition, face recognition with occlusions, and illumination invariant face recognition, under "closed set", "semi-open set", and "open set" recognition scenarios, that our proposed FKDA significantly outperforms traditional linear discriminant subspace learning methods as well as five other competing algorithms. HighlightsSolve small-sample-size problem in LDA, UDP, LPP using FKT formulation.Can work with high dimensional data without inverting any scatter matrices.Finds optimal projection direction vectors that are orthogonal.Finds exact solutions to the objective in the form of trace ratio.Improvement in unconstrained face recognition scenarios.
---
paper_title: The open-set problem in acoustic scene classification

paper_content:

Acoustic scene classification (ASC) has attracted growing research interest in recent years. Whereas the previous work has investigated closed-set classification scenarios, the predominant ASC application is open-set in nature. The contributions of the paper are (i) the first investigation of ASC in an open-set scenario, (ii) the formulation of open-set ASC as a detection problem, (iii) a classifier tailored to the open-set scenario and (iv) a new assessment protocol and metric. Experiments show that, despite the challenge of open-set ASC, reliable performance is achieved with the support vector data description classifier for varying levels of openness.
---
paper_title: Towards Open World Recognition

paper_content:

With the of advent rich classification models and high computational power visual recognition systems have found many operational applications. Recognition in the real world poses multiple challenges that are not apparent in controlled lab environments. The datasets are dynamic and novel categories must be continuously detected and then added. At prediction time, a trained system has to deal with myriad unseen categories. Operational systems require minimal downtime, even to learn. To handle these operational issues, we present the problem of Open World Recognition and formally define it. We prove that thresholding sums of monotonically decreasing functions of distances in linearly transformed feature space can balance “open space risk” and empirical risk. Our theory extends existing algorithms for open world recognition. We present a protocol for evaluation of open world recognition systems. We present the Nearest Non-Outlier (NNO) algorithm that evolves model efficiently, adding object categories incrementally while detecting outliers and managing open space risk. We perform experiments on the ImageNet dataset with 1.2M+ images to validate the effectiveness of our method on large scale visual recognition tasks. NNO consistently yields superior results on open world recognition.
---
paper_title: The Extreme Value Machine

paper_content:

It is often desirable to be able to recognize when inputs to a recognition function learned in a supervised manner correspond to classes unseen at training time. With this ability, new class labels could be assigned to these inputs by a human operator, allowing them to be incorporated into the recognition function—ideally under an efficient incremental update mechanism. While good algorithms that assume inputs from a fixed set of classes exist, e.g. , artificial neural networks and kernel machines, it is not immediately obvious how to extend them to perform incremental learning in the presence of unknown query classes. Existing algorithms take little to no distributional information into account when learning recognition functions and lack a strong theoretical foundation. We address this gap by formulating a novel, theoretically sound classifier—the Extreme Value Machine (EVM). The EVM has a well-grounded interpretation derived from statistical Extreme Value Theory (EVT), and is the first classifier to be able to perform nonlinear kernel-free variable bandwidth incremental learning. Compared to other classifiers in the same deep network derived feature space, the EVM is accurate and efficient on an established benchmark partition of the ImageNet dataset.
---
paper_title: Specialized Support Vector Machines for Open-set Recognition

paper_content:

Often, when dealing with real-world recognition problems, we do not need, and often cannot have, knowledge of the entire set of possible classes that might appear during operational testing. In such cases, we need to think of robust classification methods able to deal with the"unknown"and properly reject samples belonging to classes never seen during training. Notwithstanding, almost all existing classifiers to date were mostly developed for the closed-set scenario, i.e., the classification setup in which it is assumed that all test samples belong to one of the classes with which the classifier was trained. In the open-set scenario, however, a test sample can belong to none of the known classes and the classifier must properly reject it by classifying it as unknown. In this work, we extend upon the well-known Support Vector Machines (SVM) classifier and introduce the Specialized Support Vector Machines (SSVM), which is suitable for recognition in open-set setups. SSVM balances the empirical risk and the risk of the unknown and ensures that the region of the feature space in which a test sample would be classified as known (one of the known classes) is always bounded, ensuring a finite risk of the unknown. In this work, we also highlight the properties of the SVM classifier related to the open-set scenario, and provide necessary and sufficient conditions for an RBF SVM to have bounded open-space risk.
---
paper_title: Incremental and Distributed Learning with Support Vector Machines

paper_content:

Due to the increase in the amount of data gathered every day in the real world problems (e.g., bioinformatics), there is a need for inductive learning algorithms that can incrementally process large amounts of data that is being accumulated over time in physically distributed, autonomous data repositories. In the incremental setting, the learner gradually refines a hypothesis (or a set of hypotheses) as new data become available. Because of the large volume of data involved, it may not be practical to store and access the entire dataset during learning. Thus, the learner does not have access to data that has been encountered at a previous time. Learning in the distributed setting can be defined in a similar fashion. An incremental or distributed learning algorithm is said to be exact if it gives the same results as those obtained by batch learning (i.e., when the entire dataset is accessible to the learning algorithm during learning). We explore exact distributed and incremental learning algorithms that are variants and extensions of the support vector machine (SVM) family of learning algorithms. For the sake of simplicity, suppose that we have two data sets and , and we want to learn from them in an incremental setting using SVM. A naive approach (Syed, Liu & Sung, 1999) works as follows: 1. Apply the SVM algorithm to and generate a set of support vectors
---
paper_title: On optimum recognition error and reject tradeoff

paper_content:

The performance of a pattern recognition system is characterized by its error and reject tradeoff. This paper describes an optimum rejection rule and presents a general relation between the error and reject probabilities and some simple properties of the tradeoff in the optimum recognition system. The error rate can be directly evaluated from the reject function. Some practical implications of the results are discussed. Examples in normal distributions and uniform distributions are given.
---
paper_title: Finding the Unknown: Novelty Detection with Extreme Value Signatures of Deep Neural Activations

paper_content:

Achieving or even surpassing human-level accuracy became recently possible in a variety of application scenarios due to the rise of convolutional neural networks (CNNs) trained from large datasets. However, solving supervised visual recognition tasks by discriminating among known categories is only one side of the coin. In contrast to this, novelty detection is still an unsolved task where instances of yet unknown categories need to be identified. Therefore, we propose to leverage the powerful discriminative nature of CNNs to novelty detection tasks by investigating class-specific activation patterns. More precisely, we assume that a semantic category can be described by its extreme value signature, that specifies which dimensions of deep neural activations have largest values. By following this intuition, we show that already a small number of high-valued dimensions allows to separate known from unknown categories. Our approach is simple, intuitive, and can be easily put on top of CNNs trained for vanilla classification tasks. We empirically validate the benefits of our approach in terms of accuracy and speed by comparing it against established methods in a variety of novelty detection tasks derived from ImageNet. Finally, we show that visualizing extreme value signatures allows to inspect class-specific patterns learned during training which may ultimately help to better understand CNN models.
---
paper_title: Growing a multi-class classifier with a reject option

paper_content:

In many classification problems objects should be rejected when the confidence in their classification is too low. An example is a face recognition problem where the faces of a selected group of people have to be classified, but where all other faces and non-faces should be rejected. These problems are typically solved by estimating the class densities and assigning an object to the class with the highest posterior probability. The total probability density is thresholded to detect the outliers. Unfortunately, this procedure does not easily allow for class-dependent thresholds, or for class models that are not based on probability densities but on distances. In this paper we propose a new heuristic to combine any type of one-class models for solving the multi-class classification problem with outlier rejection. It normalizes the average model output per class, instead of the more common non-linear transformation of the distances. It creates the possibility to adjust the rejection threshold per class, and also to combine class models that are not (all) based on probability densities and to add class models without affecting the boundaries of existing models. Experiments show that for several classification problems using class-specific models significantly improves the performance.
---
paper_title: Open set face recognition using transduction

paper_content:

This paper motivates and describes a novel realization of transductive inference that can address the open set face recognition task. Open set operates under the assumption that not all the test probes have mates in the gallery. It either detects the presence of some biometric signature within the gallery and finds its identity or rejects it, i.e., it provides for the "none of the above" answer. The main contribution of the paper is open set TCM-kNN (transduction confidence machine-k nearest neighbors), which is suitable for multiclass authentication operational scenarios that have to include a rejection option for classes never enrolled in the gallery. Open set TCM-kNN, driven by the relation between transduction and Kolmogorov complexity, provides a local estimation of the likelihood ratio needed for detection tasks. We provide extensive experimental data to show the feasibility, robustness, and comparative advantages of open set TCM-kNN on open set identification and watch list (surveillance) tasks using challenging FERET data. Last, we analyze the error structure driven by the fact that most of the errors in identification are due to a relatively small number of face patterns. Open set TCM-kNN is shown to be suitable for PSEI (pattern specific error inhomogeneities) error analysis in order to identify difficult to recognize faces. PSEI analysis improves biometric performance by removing a small number of those difficult to recognize faces responsible for much of the original error in performance and/or by using data fusion.
---
paper_title: Novelty detection and multi-class classification in power distribution voltage waveforms

paper_content:

Accurate classification of events in waveforms from electrical distribution networks.Novelty detection: dynamic identification of new classes of events.SVDD using negative examples and maximal margin separation: better generalization.Experiments using real data: significant improvements in classification accuracy.Direct application as part of tools to assist mitigation processes in power utilities. The automatic analysis of electrical waveforms is a recurring subject in the power system sector worldwide. In this sense, the idea of this paper is to present an original approach for automatic classification of voltage waveforms in electrical distribution networks. It includes both the classification of the waveforms in multiple known classes, and the detection of new waveforms (novelties) that are not available during the training stage. The classification method, based on the Support Vector Data Description (SVDD), has a suitable formulation for this task, because it is capable of fitting a model on a relatively small set of examples, which may also include negative examples (patterns from other known classes or even novelties), with maximal margin separation. The results obtained on both simulated and real world data demonstrate the ability of the method to identify novelties and to classify known examples correctly. The method finds application in the mitigation process of emergencies normally performed by power utilities' maintenance and protection engineers, which requires fast and accurate event cause identification.
---
paper_title: An optimum character recognition system using decision functions

paper_content:

The character recognition problem, usually resulting from characters being corrupted by printing deterioration and/or inherent noise of the devices, is considered from the viewpoint of statistical decision theory. The optimization consists of minimizing the expected risk for a weight function which is preassigned to measure the consequences of system decisions As an alternative minimization of the error rate for a given rejection rate is used as the critenon. The optimum recogition is thus obtained. The optimum system consists of a conditional-probability densisities computer; character channels, one for each character; a rejection channel; and a comparison network. Its precise structure and and ultimate performance depend essentially upon the signals and noise structure. Explicit examples for an additive Gaussian noise and a ``cosine'' noise are presented. Finally, an error-free recognition system and a possible criterion to measure the character style and deteriortation are presented.
---
paper_title: Modular ensembles for one-class classification based on density analysis

paper_content:

One-Class Classification (OCC) is an important machine learning task. It studies a special classification problem that training samples from only one class, named target class, are available or reliable. Recently, various OCC algorithms have been proposed, however many of them do not adequately deal with multi-modality, multi-density, the noise and arbitrarily shaped distributions of the target class. In this paper, we propose a novel Density Based Modular Ensemble One-class Classifier (DBM-EOC) algorithm which is motivated by density analysis, divide-and-conquer method and ensemble learning. DBM-EOC first performs density analysis on training samples to obtain a minimal spanning tree using density characteristics of the target class. On this basis, DBM-EOC automatically identifies clusters, multi-density distributions and the noise in training samples using extreme value analysis. Then target samples are categorized into several groups called Local Dense Subset (LDS). Samples in each LDS are close to each other and their local densities are similar. A simple base OCC model e.g. the Gaussian estimator is built for each LDS afterwards. Finally all the base classifiers are modularly aggregated to construct the DBM-EOC model. We experimentally evaluate DBM-EOC with 6 state-of-art OCC algorithms on 5 synthetic datasets, 18 UCI benchmark datasets and the MNIST dataset. The results show that DBM-EOC outperforms other competitors in majority cases especially when the datasets are multi-modality, multi-density or noisy. We propose a modular ensemble OCC algorithm DBM-EOC based on density analysis.We analyze peculiarities of the target class which are crucial for OCC.DBM-EOC obtains a tree structure of the target class considering density.DBM-EOC can automatically detect clusters and remove noise samples.DBM-EOC solves OCC problems with the divide-and-conquer method.
---
paper_title: Local Novelty Detection in Multi-class Recognition Problems

paper_content:

In this paper, we propose using local learning for multiclass novelty detection, a framework that we call local novelty detection. Estimating the novelty of a new sample is an extremely challenging task due to the large variability of known object categories. The features used to judge on the novelty are often very specific for the object in the image and therefore we argue that individual novelty models for each test sample are important. Similar to human experts, it seems intuitive to first look for the most related images thus filtering out unrelated data. Afterwards, the system focuses on discovering similarities and differences to those images only. Therefore, we claim that it is beneficial to solely consider training images most similar to a test sample when deciding about its novelty. Following the principle of local learning, for each test sample a local novelty detection model is learned and evaluated. Our local novelty score turns out to be a valuable indicator for deciding whether the sample belongs to a known category from the training set or to a new, unseen one. With our local novelty detection approach, we achieve state-of-the-art performance in multi-class novelty detection on two popular visual object recognition datasets, Caltech-256 and Image Net. We further show that our framework: (i) can be successfully applied to unknown face detection using the Labeled-Faces-in-the-Wild dataset and (ii) outperforms recent work on attribute-based unfamiliar class detection in fine-grained recognition of bird species on the challenging CUB-200-2011 dataset.
---
paper_title: Support vector machines with a reject option

paper_content:

This paper studies $\ell_1$ regularization with high-dimensional features for support vector machines with a built-in reject option (meaning that the decision of classifying an observation can be withheld at a cost lower than that of misclassification). The procedure can be conveniently implemented as a linear program and computed using standard software. We prove that the minimizer of the penalized population risk favors sparse solutions and show that the behavior of the empirical risk minimizer mimics that of the population risk minimizer. We also introduce a notion of classification complexity and prove that our minimizers adapt to the unknown complexity. Using a novel oracle inequality for the excess risk, we identify situations where fast rates of convergence occur.
---
paper_title: RO-SVM: Support Vector Machine with Reject Option for Image Categorization

paper_content:

When applying Multiple Instance Learning (MIL) for image categorization, an image is treated as a bag containing a number of instances, each representing a region inside the image. The categorization of this image is determined by the labels of these instances, which are not specified in the training data-set. Hence, these instance labels are needed to be estimated together with the classifier. To improve classification reliability, we propose in this paper a new Support Vector Machine approach by incorporating a reject option, named RO-SVM to determine the instance labels, and the rejection region during the training phase simultaneously. Our approach can also be easily extended to solve multi-class classification problems. Experimental results demonstrate that higher categorization accuracy can be achieved with our RO-SVM method, comparing to approaches that do not exclude uninformative image patches. Our method is able to produce results comparable even with few training samples.
---
paper_title: Support Vector Machines with Embedded Reject Option

paper_content:

In this paper, the problem of implementing the reject option in support vector machines (SVMs) is addressed. We started by observing that methods proposed so far simply apply a reject threshold to the outputs of a trained SVM. We then showed that, under the framework of the structural risk minimisation principle, the rejection region must be determined during the training phase of a classifier. By applying this concept, and by following Vapnik's approach, we developed a maximum margin classifier with reject option. This led us to a SVM whose rejection region is determined during the training phase, that is, a SVM with embedded reject option. To implement such a SVM, we devised a novel formulation of the SVM training problem and developed a specific algorithm to solve it. Preliminary results on a character recognition problem show the advantages of the proposed SVM in terms of the achievable error-reject trade-off.
---
paper_title: Assumed density filtering methods for learning Bayesian neural networks

paper_content:

Buoyed by the success of deep multilayer neural networks, there is renewed interest in scalable learning of Bayesian neural networks. Here, we study algorithms that utilize recent advances in Bayesian inference to efficiently learn distributions over network weights. In particular, we focus on recently proposed assumed density filtering based methods for learning Bayesian neural networks – Expectation and Probabilistic backpropagation. Apart from scaling to large datasets, these techniques seamlessly deal with non-differentiable activation functions and provide parameter (learning rate, momentum) free learning. In this paper, we first rigorously compare the two algorithms and in the process develop several extensions, including a version of EBP for continuous regression problems and a PBP variant for binary classification. Next, we extend both algorithms to deal with multiclass classification and count regression problems. On a variety of diverse real world benchmarks, we find our extensions to be effective, achieving results competitive with the state-of-the-art.
---
paper_title: Reducing Network Agnostophobia

paper_content:

Agnostophobia, the fear of the unknown, can be experienced by deep learning engineers while applying their networks to real-world applications. Unfortunately, network behavior is not well defined for inputs far from a networks training set. In an uncontrolled environment, networks face many instances that are not of interest to them and have to be rejected in order to avoid a false positive. This problem has previously been tackled by researchers by either a) thresholding softmax, which by construction cannot return "none of the known classes", or b) using an additional background or garbage class. In this paper, we show that both of these approaches help, but are generally insufficient when previously unseen classes are encountered. We also introduce a new evaluation metric that focuses on comparing the performance of multiple approaches in scenarios where such unseen classes or unknowns are encountered. Our major contributions are simple yet effective Entropic Open-Set and Objectosphere losses that train networks using negative samples from some classes. These novel losses are designed to maximize entropy for unknown inputs while increasing separation in deep feature space by modifying magnitudes of known and unknown samples. Experiments on networks trained to classify classes from MNIST and CIFAR-10 show that our novel loss functions are significantly better at dealing with unknown inputs from datasets such as Devanagari, NotMNIST, CIFAR-100, and SVHN.
---
paper_title: SSD: Single Shot MultiBox Detector

paper_content:

We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For \(300 \times 300\) input, SSD achieves 74.3 % mAP on VOC2007 test at 59 FPS on a Nvidia Titan X and for \(512 \times 512\) input, SSD achieves 76.9 % mAP, outperforming a comparable state of the art Faster R-CNN model. Compared to other single stage methods, SSD has much better accuracy even with a smaller input image size. Code is available at https://github.com/weiliu89/caffe/tree/ssd.
---
paper_title: The Extreme Value Machine

paper_content:

It is often desirable to be able to recognize when inputs to a recognition function learned in a supervised manner correspond to classes unseen at training time. With this ability, new class labels could be assigned to these inputs by a human operator, allowing them to be incorporated into the recognition function—ideally under an efficient incremental update mechanism. While good algorithms that assume inputs from a fixed set of classes exist, e.g. , artificial neural networks and kernel machines, it is not immediately obvious how to extend them to perform incremental learning in the presence of unknown query classes. Existing algorithms take little to no distributional information into account when learning recognition functions and lack a strong theoretical foundation. We address this gap by formulating a novel, theoretically sound classifier—the Extreme Value Machine (EVM). The EVM has a well-grounded interpretation derived from statistical Extreme Value Theory (EVT), and is the first classifier to be able to perform nonlinear kernel-free variable bandwidth incremental learning. Compared to other classifiers in the same deep network derived feature space, the EVM is accurate and efficient on an established benchmark partition of the ImageNet dataset.
---
paper_title: Adversarial Robustness: Softmax versus Openmax

paper_content:

Deep neural networks (DNNs) provide state-of-the-art results on various tasks and are widely used in real world applications. However, it was discovered that machine learning models, including the best performing DNNs, suffer from a fundamental problem: they can unexpectedly and confidently misclassify examples formed by slightly perturbing otherwise correctly recognized inputs. Various approaches have been developed for efficiently generating these so-called adversarial examples, but those mostly rely on ascending the gradient of loss. In this paper, we introduce the novel logits optimized targeting system (LOTS) to directly manipulate deep features captured at the penultimate layer. Using LOTS, we analyze and compare the adversarial robustness of DNNs using the traditional Softmax layer with Openmax, which was designed to provide open set recognition by defining classes derived from deep representations, and is claimed to be more robust to adversarial perturbations. We demonstrate that Openmax provides less vulnerable systems than Softmax to traditional attacks, however, we show that it can be equally susceptible to more sophisticated adversarial generation techniques that directly work on deep representations.
---
paper_title: Towards Reaching Human Performance in Pedestrian Detection

paper_content:

Encouraged by the recent progress in pedestrian detection, we investigate the gap between current state-of-the-art methods and the “perfect single frame detector”. We enable our analysis by creating a human baseline for pedestrian detection (over the Caltech pedestrian dataset). After manually clustering the frequent errors of a top detector, we characterise both localisation and background-versus-foreground errors. To address localisation errors we study the impact of training annotation noise on the detector performance, and show that we can improve results even with a small portion of sanitised training data. To address background/foreground discrimination, we study convnets for pedestrian detection, and discuss which factors affect their performance. Other than our in-depth analysis, we report top performance on the Caltech pedestrian dataset, and provide a new sanitised set of training and test annotations.
---
paper_title: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

paper_content:

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.
---
paper_title: Figure of Merit Training for Detection and Spotting

paper_content:

Spotting tasks require detection of target patterns from a background of richly varied non-target inputs. The performance measure of interest for these tasks, called the figure of merit (FOM), is the detection rate for target patterns when the false alarm rate is in an acceptable range. A new approach to training spotters is presented which computes the FOM gradient for each input pattern and then directly maximizes the FOM using backpropagation. This eliminates the need for thresholds during training. It also uses network resources to model Bayesian a posteriori probability functions accurately only for patterns which have a significant effect on the detection accuracy over the false alarm rate of interest. FOM training increased detection accuracy by 5 percentage points for a hybrid radial basis function (RBF) - hidden Markov model (HMM) wordspotter on the credit-card speech corpus.
---
paper_title: Intriguing properties of neural networks

paper_content:

Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. ::: First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. ::: Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
---
paper_title: The Pascal Visual Object Classes (VOC) Challenge

paper_content:

The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension.
---
paper_title: Microsoft COCO: Common Objects in Context

paper_content:

We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.
---
paper_title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

paper_content:

Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study [30] revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call “fooling images” (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.
---
paper_title: Confidence Prediction for Lexicon-Free OCR

paper_content:

Having a reliable accuracy score is crucial for real world applications of OCR, since such systems are judged by the number of false readings. Lexicon-based OCR systems, which deal with what is essentially a multi-class classification problem, often employ methods explicitly taking into account the lexicon, in order to improve accuracy. However, in lexicon-free scenarios, filtering errors requires an explicit confidence calculation. In this work we show two explicit confidence measurement techniques, and show that they are able to achieve a significant reduction in misreads on both standard benchmarks and a proprietary dataset.
---
paper_title: Fast R-CNN

paper_content:

This paper proposes a Fast Region-based Convolutional Network method (Fast R-CNN) for object detection. Fast R-CNN builds on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Fast R-CNN employs several innovations to improve training and testing speed while also increasing detection accuracy. Fast R-CNN trains the very deep VGG16 network 9x faster than R-CNN, is 213x faster at test-time, and achieves a higher mAP on PASCAL VOC 2012. Compared to SPPnet, Fast R-CNN trains VGG16 3x faster, tests 10x faster, and is more accurate. Fast R-CNN is implemented in Python and C++ (using Caffe) and is available under the open-source MIT License at https://github.com/rbgirshick/fast-rcnn.
---
paper_title: Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

paper_content:

Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.
---
paper_title: Towards Open Set Deep Networks

paper_content:

Deep networks have produced significant gains for various visual recognition problems, leading to high impact academic and commercial applications. Recent work in deep networks highlighted that it is easy to generate images that humans would never classify as a particular object class, yet networks classify such images high confidence as that given class - deep network are easily fooled with images humans do not consider meaningful. The closed set nature of deep networks forces them to choose from one of the known classes leading to such artifacts. Recognition in the real world is open set, i.e. the recognition system should reject unknown/unseen classes at test time. We present a methodology to adapt deep networks for open set recognition, by introducing a new model layer, OpenMax, which estimates the probability of an input being from an unknown class. A key element of estimating the unknown probability is adapting Meta-Recognition concepts to the activation patterns in the penultimate layer of the network. OpenMax allows rejection of "fooling" and unrelated open set images presented to the system; OpenMax greatly reduces the number of obvious errors made by a deep network. We prove that the OpenMax concept provides bounded open space risk, thereby formally providing an open set recognition solution. We evaluate the resulting open set deep networks using pre-trained networks from the Caffe Model-zoo on ImageNet 2012 validation data, and thousands of fooling and open set images. The proposed OpenMax model significantly outperforms open set recognition accuracy of basic deep networks as well as deep networks with thresholding of SoftMax probabilities.
---
paper_title: A Discriminative Feature Learning Approach for Deep Face Recognition

paper_content:

Convolutional neural networks (CNNs) have been widely used in computer vision community, significantly improving the state-of-the-art. In most of the available CNNs, the softmax loss function is used as the supervision signal to train the deep model. In order to enhance the discriminative power of the deeply learned features, this paper proposes a new supervision signal, called center loss, for face recognition task. Specifically, the center loss simultaneously learns a center for deep features of each class and penalizes the distances between the deep features and their corresponding class centers. More importantly, we prove that the proposed center loss function is trainable and easy to optimize in the CNNs. With the joint supervision of softmax loss and center loss, we can train a robust CNNs to obtain the deep features with the two key learning objectives, inter-class dispension and intra-class compactness as much as possible, which are very essential to face recognition. It is encouraging to see that our CNNs (with such joint supervision) achieve the state-of-the-art accuracy on several important face recognition benchmarks, Labeled Faces in the Wild (LFW), YouTube Faces (YTF), and MegaFace Challenge. Especially, our new approach achieves the best results on MegaFace (the largest public domain face benchmark) under the protocol of small training set (contains under 500000 images and under 20000 persons), significantly improving the previous results and setting new state-of-the-art for both face recognition and face verification tasks.
---
paper_title: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

paper_content:

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.
---

Title: Learning and the Unknown: Surveying Steps toward Open World Recognition

Section 1: Introduction
Description 1: Introduce the concept of open world recognition and its importance in current machine learning research.

Section 2: Formalizing Open Set Recognition
Description 2: Discuss the formalization of open set recognition, including the definition, mathematical models, and key properties.

Section 3: Approaches that sometimes solve OSR
Description 3: Analyze various approaches and discuss why some algorithms may partially address OSR while others fall short.

Section 4: Open Set Deep Networks
Description 4: Examine the challenges and current solutions related to open set recognition in deep learning frameworks.

Section 5: Conclusion
Description 5: Summarize the key points, the progress made, and emphasize the ongoing challenges and future directions in the field of open world recognition.