We introduce a novel geometric perspective and unsupervised model augmentation framework for transforming traditional deep (convolutional) neural networks into adversarially robust classifiers. Class-conditional probability densities based on Bayesian nonparametric mixtures of factor analyzers (BNP-MFA) over the input space are used to design soft decision labels for feature to label isometry. Classconditional distributions over features are also learned using BNP-MFA to develop plug-in maximum a posterior (MAP) classifiers to replace the traditional multinomial logistic softmax classification layers. This novel unsupervised augmented framework, which we call geometrically robust networks (GRN), is applied to CIFAR-10, CIFAR-100, and to Radio-ML (a time series dataset for radio modulation recognition). We demonstrate the robustness of GRN models to adversarial attacks from fast gradient sign method, Carlini-Wagner, and projected gradient descent. DeepConvNets are already prevalent in speech, vision, self-driving cars, biometrics, and robotics. However, they possess discontinuities that are easy targets for attacks as evidenced in dozens of papers (see BID7 BID15 and references therein). Adversarial images can be made to be robust to translation, scale, and rotation BID0 . Adversarial attacks have also been applied to deep reinforcement learning BID9 BID10 and speech recognition BID3 . In this work we will also consider attacks on automatic modulation recognition using deep convolutional networks BID17 . Previous work in creating adversarially robust deep neural network classifiers includes robust optimization with saddle point formulations BID13 , adversarial training (see e.g., BID11 ), ensemble adversarial training BID24 , defensive distillation BID19 , and use of detector-reformer networks BID14 . Defensive distillation has been found to be an insufficient defense BID1 and MagNet of BID14 was also shown to be defeatable in BID4 . A summary of the attacks and defenses from the NIPS 2017 competition on adversarial attack and defense can be found in .In this paper we propose a statistical geometric model augmentation approach to designing robust neural networks. We argue that signal representations involving projections onto lower-dimensional subspaces lower mean square error distortion. We implement a statistical union of subspaces learned using a mixture of factor analyzers to create the auxiliary signal space structural information that neural networks can use to improve robustness. We use the geometry of the input space to create unsupervised soft probabilistic decision labels to replace traditional hard one-hot encoded label vectors. We also use the geometry of the feature space (after soft-decision supervised training) to create accurate class-conditional probability density estimates for MAP classifiers (to replace neural network classification layers). We call this unsupervised geometric augmentation framework geometrically robust networks (GRN). The main contributions of this paper are:1. Geometric analysis of problems with current neural networks.2. A novel soft decision label coding framework using unsupervised statistical-geometric union of subspace learning.3. Maximum a posteriori classification framework based on class-conditional feature vector density estimation.The rest of this paper is organized as follows. In Section 2 we analyze neural networks from a geometric vantage point and recommend solution pathways for overcoming adversarial brittleness. In Section 3 we describe the full details of the proposed geometrically robust network design framework. We give experimental results on two datasets and three attacks in Section 4 and conclude in Section 5. We have demonstrated that geometrical statistically augmented neural network models can achieve state-of-the-art robustness on CIFAR-10 under three different adversarial attack methods. We hope that this work will be the start of further investigation into the idea of using geometrically centered unsupervised learning methods to assist in making deep learning models robust, not only to adversarial noise but to all types of noise. There is more work that could be done to understand the best way to engineer soft decision labels given auxiliary data models. We need to also understand if the training algorithms themselves can be directly manipulated to incorporate outside structural data models.A main selling point of Bayesian nonparametrics has been that the complexity of the model can grow as more data is observed. However, the current training algorithm for the BNP-MFA model is Gibbs sampling, which fails to scale to massive data sets. Stochastic variational inference BID8 has been introduced as one such way to perform variational inference for massive or streaming data sets. We are currently working to cast the BNP-MFA into a stochastic variational framework so that the GRN model can be extended to very large (or even streaming) datasets. Figure 4: Network specification and performance results for proposed geometrically robust networks applied to the Radio-ML dataset (modulation recognition over 11 modulation formats).
We develop a statistical-geometric unsupervised learning augmentation framework for deep neural networks to make them robust to adversarial attacks. Transfroms traditional deep neural networks into adversarial robust calssifiers using GRNs Proposes a defense based on class-conditional feature distributions to turn deep neural netwroks into robust classifiers