Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classification. Little studies are available that compare the effectiveness of these approaches for character based text classification with each other. In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm). Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overfitting. Malware is software that infects computers in order to perform unauthorized malicious activities. In order to successfully achieve its goals, the malware needs to be able to connect to a command and control (C&C) center. To this end, both the controller behind the C&C center (hereafter called botmaster) and the malware on the infected machines can run a Domain Generation Algorithm (DGA) that generates hundreds or even thousands of domains automatically. The malware then attempts at resolving each one of these domains with its local DNS server. The botmaster will have registered one or a few of these automatically generated domains. For these domains that have been actually registered, the malware will obtain a valid IP address and will be able to communicate with the C&C center.The binary text classification task that we address in this paper is: given a domain name string as input, classify it as either malicious, i.e. generated by a DGA, or as benign. Deep neural networks have recently appeared in the literature on DGA detection ; BID8 ; BID15 . They significantly outperform traditional machine learning methods in accuracy, at the price of increasing the complexity of training the model and requiring larger datasets. Independent of the work on deep networks for DGA detection, other deep learning approaches for character based text classification have recently been proposed, including deep neural network architectures designed for processing and classification of tweets BID2 ; BID11 ) as well as general natural language text BID16 ). No systematic study is available that compares the predictive accuracy of all these different character based deep learning architectures, leaving one to wonder which one works best for DGA detection.To answer this open question, in this paper we compare the performance of five different deep learning architectures for character based text classification (see TAB0 ) for the problem of detecting DGAs. They all rely on character-level embeddings, and they all use a deep learning architecture based on convolutional neural network (CNN) layers, recurrent neural network (RNN) layers, or a combination of both. Our most important finding is that for DGA detection, which can be thought of as classification of short character strings, despite of vast differences in the deep network architectures, there is remarkably little difference among the methods in terms of accuracy and false positive rates, while they all comfortably outperform a random forest trained on human engineered features. This finding is of practical value for the design of deep neural network based classifiers for short text classification in industry and academia: it provides evidence that one can select an architecture that BID16 is faster to train, without loss of accuracy. In the context of DGA detection, optimizing the training time is of particular importance, as the models need to be retrained on a regular basis to stay current with respect to new, emerging malware. DGA detection, i.e. the classification task of distinguishing between benign domain names and those generated by malware (Domain Generation Algorithms), has become a central topic in information security. In this paper we have compared five different deep neural network architectures that perform this classification task based purely on the domain name string, given as a raw input signal at character level. All five models, i.e. two RNN based architectures, two CNN based architectures, and one hybrid RNN/CNN architecture perform equally well, catching around 97-98% of malicious domain names against a false positive rate of 0.001. This roughly means that for every 970 malicious domain names that the deep networks catch, they flag only one benign domain name erroneously as malicious. A Random Forest based on human defined linguistic features achieves a recall of only 83% against the same 0.001 false positive rate when trained and tested on the same data that was used for the deep networks. The use of a deep neural network that automatically learns features is attractive in a cybersecurity setting because it is a lot harder to craft malware to avoid detection by a system that relies on automatically learned features instead of on human engineered features. An interesting direction for future work is to test the trained deep networks more extensively on domain names generated by new and previously unseen malware families.A KERAS CODE FOR DEEP NETWORKS main input = Input (shape=(75, ) , dtype='int32 ', name='main input') embedding = Embedding(input dim=128, output dim=128, input length =75)(main input ) lstm = LSTM(128, return sequences=False)(embedding) drop = Dropout(0.5) (lstm) output = Dense(1, activation ='sigmoid') (drop) model = Model(inputs=main input, outputs =output) model.compile( loss =' binary crossentropy ', optimizer ='adam')Listing 1: Endgame model with single LSTM layer, adapted from main input = Input (shape=(75, ) , dtype='int32 ', name='main input') embedding = Embedding(input dim=128, output dim=128, input length =75)(main input ) bi lstm = Bidirectional ( layer =LSTM(64, return sequences=False), merge mode='concat')(embedding) output = Dense(1, activation ='sigmoid') ( bi lstm ) model = Model(inputs=main input, outputs =output) model.compile( loss =' binary crossentropy ', optimizer ='adam') Listing 2: CMU model with bidirectional LSTM, adapted from BID2 main input = Input (shape=(75, ) , dtype='int32 ', name='main input') embedding = Embedding(input dim=128, output dim=128, input length =75)(main input ) conv1 = Conv1D( filters =128, kernel size =3, padding='same', strides =1)(embedding) thresh1 = ThresholdedReLU(1e−6)(conv1) max pool1 = MaxPooling1D(pool size=2, padding='same')(thresh1 ) conv2 = Conv1D( filters =128, kernel size =2, padding='same', strides =1)(max pool1) thresh2 = ThresholdedReLU(1e−6)(conv2) max pool2 = MaxPooling1D(pool size=2, padding='same')(thresh2 ) flatten = Flatten () (max pool2) fc = Dense(64)( flatten ) thresh fc = ThresholdedReLU(1e−6)(fc) drop = Dropout(0.5) ( thresh fc ) output = Dense(1, activation ='sigmoid') (drop) model = Model(inputs=main input, outputs =output) model.compile( loss =' binary crossentropy ', optimizer ='adam') Listing 3: NYU model with stacked CNN layers, adapted from BID16 def getconvmodel( self , kernel size , filters ) : model = Sequential () model.add( Conv1D( filters = filters , input shape =(128, 128), kernel size = kernel size , padding='same', activation =' relu ', strides =1)) model.add(Lambda(lambda x: K.sum(x, axis=1), output shape =( filters , ) ) ) model.add(Dropout(0.5) ) return model main input = Input (shape=(75, ) , dtype='int32 ', name='main input') embedding = Embedding(input dim=128, output dim=128, input length =75)(main input ) conv1 = getconvmodel(2, 256)(embedding) conv2 = getconvmodel(3, 256)(embedding) conv3 = getconvmodel(4, 256)(embedding) conv4 = getconvmodel(5, 256)(embedding) merged = Concatenate () ([ conv1, conv2, conv3, conv4] ) middle = Dense(1024, activation =' relu ') (merged) middle = Dropout(0.5) (middle) middle = Dense(1024, activation =' relu ') (middle) middle = Dropout(0.5) (middle) output = Dense(1, activation ='sigmoid') (middle) model = Model(inputs=main input, outputs =output) model.compile( loss =' binary crossentropy ', optimizer ='adam') Listing 4: Invincea CNN model with parallel CNN layers, adapted from BID8 main input = Input (shape=(75, ) , dtype='int32 ', name='main input') embedding = Embedding(input dim=128, output dim=128, input length =75)(main input ) conv = Conv1D( filters =128, kernel size =3, padding='same', activation =' relu ', strides =1)(embedding) max pool = MaxPooling1D(pool size=2, padding='same')(conv) encode = LSTM(64, return sequences=False) (max pool) output = Dense(1, activation ='sigmoid') (encode) model = Model(inputs=main input, outputs =output) model.compile( loss =' binary crossentropy ', optimizer ='adam') Listing 5: MIT model with a stacked CNN and LSTM layer, adapted from BID11 main input = Input (shape=(75, ) , dtype='int32 ', name='main input') embedding = Embedding(input dim=128, output dim=128, input length =75)(main input ) flatten = Flatten () (embedding) output = Dense(1, activation ='sigmoid') ( flatten ) model = Model(inputs=main input, outputs =output) print (model.summary()) model.compile( loss =' binary crossentropy ', optimizer ='adam') Listing 6: Baseline Model with only Embedding Layer main input = Input (shape=(11, ) , name='main input') dense = Dense(128, activation =' relu ') ( main input ) output = Dense(1, activation ='sigmoid') (dense) model = Model(inputs=main input, outputs =output) print (model.summary()) model.compile( loss =' binary crossentropy ', optimizer ='adam') Listing 7: MLP Model with 128 Nodes Dense Layer
A comparison of five deep neural network architectures for detection of malicious domain names shows surprisingly little difference. Authors propose using five deep architectures for the cybersecurity task of domain generation algorithm detection. Applies several NN architectures to classify url's between begign and malware related URLs. This paper proposes to automatically recognize domain names as malicious or benign by deep networks trained to directly classify the character sequence as such.