Recent advances in adversarial Deep Learning (DL) have opened up a new and largely unexplored surface for malicious attacks jeopardizing the integrity of autonomous DL systems. We introduce a novel automated countermeasure called Parallel Checkpointing Learners (PCL) to thwart the potential adversarial attacks and significantly improve the reliability (safety) of a victim DL model. The proposed PCL methodology is unsupervised, meaning that no adversarial sample is leveraged to build/train parallel checkpointing learners. We formalize the goal of preventing adversarial attacks as an optimization problem to minimize the rarely observed regions in the latent feature space spanned by a DL network. To solve the aforementioned minimization problem, a set of complementary but disjoint checkpointing modules are trained and leveraged to validate the victim model execution in parallel. Each checkpointing learner explicitly characterizes the geometry of the input data and the corresponding high-level data abstractions within a particular DL layer. As such, the adversary is required to simultaneously deceive all the defender modules in order to succeed. We extensively evaluate the performance of the PCL methodology against the state-of-the-art attack scenarios, including Fast-Gradient-Sign (FGS), Jacobian Saliency Map Attack (JSMA), Deepfool, and Carlini&WagnerL2 algorithm. Extensive proof-of-concept evaluations for analyzing various data collections including MNIST, CIFAR10, and ImageNet corroborate the effectiveness of our proposed defense mechanism against adversarial samples. Security and safety consideration is a major obstacle to the wide-scale adoption of emerging learning algorithms in sensitive scenarios, such as intelligent transportation, healthcare, and video surveillance applications BID18 ; BID5 ; BID12 ). While advanced learning technologies are essential for enabling coordination and interaction between the autonomous agents and the environment, a careful analysis of their decision reliability in the face of carefully crafted adversarial samples ; ; BID22 ; BID3 ) and thwarting their vulnerabilities are still in their infancy.Consider a traffic sign classifier employed in self-driving cars. In this setting, an adversary can carefully add imperceptible perturbation to a legitimate "stop" sign sample and fool the DL model to classify it as a "yield" sign; thus, jeopardizes the safety of the vehicle as shown in BID18 ). As such, it is highly important to reject risky adversarial samples to ensure the integrity of DL models used in autonomous systems such as unmanned vehicles and drones. In this paper, we aim to answer two open questions regarding the adversarial attacks.(i ) Why are machine learning models vulnerable to adversarial samples? Our hypothesis is that the vulnerability of neural networks to adversarial samples originates from the existence of rarely explored sub-spaces in each feature map. This phenomenon is particularly caused by the limited access to the labeled data and/or inefficiency of regularization algorithms BID30 ; BID7 ). Figure 1 provides a simple illustration of the partially explored space in a twodimensional setup. We analytically and empirically back up our hypothesis by extensive evaluations on the state-of-the-art attacks, including Fast-Gradient-Sign ), Jacobian Saliency Map Attack ), Deepfool BID22 ), and Carlini&WagnerL2 BID3 ).(ii) How can we characterize and thwart the underlying space for unsupervised model assurance as well as defend against the adversaries? A line of research has shown that there is a trade-off between the robustness of a model and its accuracy BID17 ; BID25 ). Taking this into account , instead of making a single model that is both robust and accurate, we introduce a new defense mechanism called Parallel Checkpointing Learners (PCL) . In this setting, the victim model is kept as is while separate defender modules are trained to checkpoint the data abstractions and assess the reliability of the victim's prediction. Each defender module characterizes the explored sub-space in the pertinent layer by learning the probability density function (pdf) of legitimate data points and marking the complement sub-spaces as rarely observed regions. Once such characterization is obtained , the checkpointing modules 1 evaluate the input sample in parallel with the victim model and raise alarm flags for data points that lie within the rarely explored regions ( Figure 1c ). As we demonstrate in Section 4, adversarial samples created by various attack methods mostly lie within the sub-spaces marked as partially explored sectors.We consider a white-box attack model in which the attacker knows everything about the victim model including its architecture, learning algorithm, and parameters. This threat model represents the most powerful attacker that can endanger the real-world applications. We validate the security of our proposed approach for different DL benchmarks including MNIST, CIFAR10, and a subset of ImageNet data. Based on the result of our analysis, we provide new insights on the reason behind the existence of adversarial transferability. We open-source our API to ensure ease of use by the users (the link is omitted for blind review purposes) and invite the community to attempt attacks against our provided benchmarks in the form of a challenge.The explicit contribution of this paper is as follows: (i) Devising an automated end-to-end framework for unsupervised model assurance as well as defending against the adversaries. (ii) Incepting the idea of parallel checkpointing learners to validate the legitimacy of data abstractions at each intermediate DL layer. (iii) Performing extensive proof-of-concept evaluations against state-of-the-art attack methods. (iv) Providing new insights regarding the transferability of adversarial samples in between different models. Figure 1: (a) In this example, data points (denoted by green and blue squares) can be easily separated in one-dimensional space. Having extra dimensions adds ambiguity in choosing the pertinent decision boundaries. For instance, all the shown boundaries (dashed lines) are sufficient to classify the raw data with full accuracy in two-dimensional space but are not equivalent in terms of robustness to noise. (b) The rarely explored space (region specified by diagonal striped) in a learning model leaves room for adversaries to manipulate the nuisance (non-critical) variables and mislead the model by crossing the decision boundaries. (c) In PCL methodology, a set of defender (checkpoint) modules is trained to characterize the data density distribution in the space spanned by the victim model. The defender modules are then used in parallel to checkpoint the reliability of the ultimate prediction and raise an alarm flag for risky samples. Note that the remaining adversarial samples that are not detected in this experiment are crafted from legitimate samples that are inherently hard to classify even by a human observer due to the closeness of decision boundaries corresponding to such classes. For instance, in the MNIST application, such adversarial samples mostly belong to class 5 that is misclassified to class 3 or class 4 misclassified as 9. Such misclassifications are indeed the model approximation error which is well-understood to the statistical nature of the models. As such, a more precise definition of adversarial samples DISPLAYFORM0 Figure 9: Example adversarial confusion matrix (a) without PCL defense mechanism, and (b) with PCL defense and a security parameter of (1%). (c) Example adversarial samples for which accurate detection is hard due to the closeness of decision boundaries for the corresponding classes.is extremely required to distinguish malicious samples form those that simply lie near the decision boundaries.We emphasize that the PCL defenders are trained in an unsupervised setting independent of the attack strategy, meaning that no adversarial sample is used to train the defender models. This is particularly important as it corroborates the effectiveness of the proposed countermeasure in the face of generic attack scenarios including possible future adversarial DL algorithms. Nevertheless, one might question the effectiveness of the proposed approach for adaptive attack algorithms that target the defender modules. A comprehensive study of possible adaptive attack algorithms is yet to be performed if such attacks are developed in the future. We emphasize that, thus far, we have been able to significantly thwart all the existing attacks with only one checkpoint model approximating the data distribution in the second-to-last layer of the corresponding models. Our proposed PCL methodology, however, provides a rather more generic approach that can be adapted/modified against potential future attacks by training parallel disjoint models (with diverse objectives/parameters) to further strengthen the defense.Figure 10 demonstrate how using multiple checkpoints with a negative correlation in parallel can effectively reduce the number of false alarms while increasing the detection rate of adversarial samples. In this experiment, we have considered MNIST data classification using LeNet model with 4 layers and FGS attack. The checkpoints are inserted in different layers of the pertinent neural network (first layer up to the second-to-last layer). We empirically select the mixing coefficients to aggregate the confidence of the checkpoint defenders for rejecting an incoming sample. Note that, there is a trade-off between the computational complexity (e.g., runtime overhead) of the PCL defenders and the reliability of the overall system. On the one hand, a high number of validation checkpoints increases the reliability of the systems, but it also increases the computational load as each input sample should be validated by more defender networks. On the other hand, a small number of checkpoints may degrade the defense mechanism performance by treating adversarial samples as legitimate ones. We are looking into automated techniques to customize the number of checkpoint modules and their corresponding mixing coefficients based on application data and physical constraints such as real-time analysis requirement as future work. This paper proposes a novel end-to-end methodology for characterizing and thwarting adversarial DL space. We introduce the concept of parallel checkpointing learners as a viable countermeasure to significantly reduce the risk of integrity attacks. The proposed PCL methodology explicitly characterizes statistical properties of the features within different layers of a neural network by learning a set of complementary dictionaries and corresponding probability density functions. The effectiveness of the PCL approach is evaluated against the state-of-the-art attack models including FGS, JSMA, Deepfool, and Carlini&WagnerL2. Proof-of-concept experiments for analyzing various data collections including MNIST, CIFAR10, and a subset of the ImageNet dataset corroborate successful detection of adversarial samples with relatively small false-positive rates.We devise an open-source API for the proposed countermeasure and invite the community to attempt attacks against the provided benchmarks in the form of a challenge. Table 2 presents the neural network architectures for the victim models used in each benchmark. The network for MNIST is the popular LeNet-3 architecture, the CIFAR-10 architecture is taken from BID4 , and the ImageNet model is inspired by the AlexNet architecture BID14 ). Table 2 : Baseline (victim) network architectures for evaluated benchmarks. Here, 128C3(2) denotes a convolutional layer with 128 maps and 3 × 3 filters applied with a stride of 2, MP3(2) indicates a max-pooling layer over regions of size 3 × 3 and stride of 2, and 300FC is a fully-connected layer consisting of 300 neurons. All convolution and fully connected layers (except the last layer) are followed by ReLU activation. A Softmax activation is applied to the last layer of each network. We visually evaluate the perturbed examples to determine the attack parameters (e.g., perturbation level ε and n iters ) such that the perturbations cannot be recognized by a human observer. Table 3 details the parameters used for the realization of different attack algorithms. The JSMA attack for the ImageNet benchmark is computationally expensive (e.g., it took more than 20min to generate one adversarial sample on an NVIDIA TITAN Xp GPU). As such, we could not generate the adversarial samples of this attack using the JSMA library provided by (Nicolas Papernot (2017) ). Table 3 : Details of attack algorithms for each evaluated application. The FGS method ) is characterized with a single ε parameter. The JSMA attack ) has two parameters: γ specifies the maximum percentage of perturbed features and θ denotes the value added to each selected feature. The Deepfool attack BID22 ) is characterized by the number of iterative updates, which we denote by n iters in this table. For the Carlini&WagnerL2 attack BID3 ), "C" denotes the confidence, "LR" is the learning rate, "steps" is the number of binary search steps, and "iterations" stands for the maximum number of iterations.
Devising unsupervised defense mechanisms against adversarial attacks is crucial to ensure the generalizability of the defense.  This paper presents a method for detecting adversarial examples in a deep learning classification setting This paper presents an unsupervised method for detecting adversarial examples of neural networks.