Network anomaly detection research: a survey
---
paper_title: Review: Intrusion detection system: A comprehensive review

paper_content:

With the increasing amount of network throughput and security threat, the study of intrusion detection systems (IDSs) has received a lot of attention throughout the computer science field. Current IDSs pose challenges on not only capricious intrusion categories, but also huge computational power. Though there is a number of existing literatures to IDS issues, we attempt to give a more elaborate image for a comprehensive review. Through the extensive survey and sophisticated organization, we propose the taxonomy to outline modern IDSs. In addition, tables and figures we summarized in the content contribute to easily grasp the overall picture of IDSs.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: Network Anomaly Detection: Methods, Systems and Tools

paper_content:

Network anomaly detection is an important and dynamic research area. Many network intrusion detection methods and systems (NIDS) have been proposed in the literature. In this paper, we provide a structured and comprehensive overview of various facets of network anomaly detection so that a researcher can become quickly familiar with every aspect of network anomaly detection. We present attacks normally encountered by network intrusion detection systems. We categorize existing network anomaly detection methods and systems based on the underlying computational techniques used. Within this framework, we briefly describe and compare a large number of network anomaly detection methods and systems. In addition, we also discuss tools that can be used by network defenders and datasets that researchers in network anomaly detection can use. We also highlight research directions in network anomaly detection.
---
paper_title: Wireless Anomaly Detection Based on IEEE 802.11 Behavior Analysis

paper_content:

Wireless communication networks are pervading every aspect of our lives due to their fast, easy, and inexpensive deployment. They are becoming ubiquitous and have been widely used to transfer critical information, such as banking accounts, credit cards, e-mails, and social network credentials. The more pervasive the wireless technology is going to be, the more important its security issue will be. Whereas the current security protocols for wireless networks have addressed the privacy and confidentiality issues, there are unaddressed vulnerabilities threatening their availability and integrity (e.g., denial of service, session hijacking, and MAC address spoofing attacks). In this paper, we describe an anomaly based intrusion detection system for the IEEE 802.11 wireless networks based on behavioral analysis to detect deviations from normal behaviors that are triggered by wireless network attacks. Our anomaly behavior analysis of the 802.11 protocols is based on monitoring the n-consecutive transitions of the protocol state machine. We apply sequential machine learning techniques to model the n-transition patterns in the protocol and characterize the probabilities of these transitions being normal. We have implemented several experiments to evaluate our system performance. By cross validating the system over two different wireless channels, we have achieved a low false alarm rate (<0.1%). We have also evaluated our approach against an attack library of known wireless attacks and has achieved more than 99% detection rate.
---
paper_title: Estimating Traffic and Anomaly Maps via Network Tomography

paper_content:

Mapping origin-destination (OD) network traffic is pivotal for network management and proactive security tasks. However, lack of sufficient flow-level measurements as well as potential anomalies pose major challenges towards this goal. Leveraging the spatiotemporal correlation of nominal traffic, and the sparse nature of anomalies, this paper brings forth a novel framework to map out nominal and anomalous traffic, which treats jointly important network monitoring tasks including traffic estimation, anomaly detection, and traffic interpolation. To this end, a convex program is first formulated with nuclear and $\ell _{1}$ -norm regularization to effect sparsity and low rank for the nominal and anomalous traffic with only the link counts and a small subset of OD-flow counts. Analysis and simulations confirm that the proposed estimator can exactly recover sufficiently low-dimensional nominal traffic and sporadic anomalies so long as the routing paths are sufficiently “spread-out” across the network, and an adequate amount of flow counts are randomly sampled. The results offer valuable insights about data acquisition strategies and network scenaria giving rise to accurate traffic estimation. For practical networks where the aforementioned conditions are possibly violated, the inherent spatiotemporal traffic patterns are taken into account by adopting a Bayesian approach along with a bilinear characterization of the nuclear and $\ell _{1}$ norms. The resultant nonconvex program involves quadratic regularizers with correlation matrices, learned systematically from (cyclo)stationary historical data. Alternating-minimization based algorithms with provable convergence are also developed to procure the estimates. Insightful tests with synthetic and real Internet data corroborate the effectiveness of the novel schemes.
---
paper_title: Post-Deployment Anomaly Detection and Diagnosis in Networked Embedded Systems by Program Profiling and Symptom Mining

paper_content:

Detecting and diagnosing anomalies in networked embedded systems like sensor networks is a very difficult task, due to the variable workloads and severe resource constraints. In this paper, we focus on how to aid bug diagnosis after the system has been deployed. We notice that most node-level debugging tools can provide detailed program information inside the node but fail to detect when and where a problem occurs in the network. On the other hand, most network-level diagnosis tools can effectively detect a problem from the network but fail to narrow down the problem within the node because they lack detailed program information. To close the gap, we propose D2, a new method for post-deployment anomaly detection and diagnosis in networked embedded systems by combining program profiling and symptom mining. D2 employs binary instrumentation to perform lightweight function count profiling. Based on the statistics, D2 uses PCA (Principal Component Analysis) based approach for automatically detecting network anomalies. Compared with previous methods, D2 is able to point programmers closer to the most likely causes by a novel approach combining statistical tests and program call graph analysis. We implement our method based on TinyOS 2.1.1 and evaluate its effectiveness by case studies in the development of a working sensor network. Results show that our method can aid programmers to diagnose problems quickly in real-world sensor network systems, and at the same time, incurs an acceptable overhead to the running system.
---
paper_title: Anomaly Detection Based on LRD Behavior Analysis of Decomposed Control and Data Planes Network Traffic Using SOSS and FARIMA Models

paper_content:

The detection of anomalies in network traffic, such as low volume attacks and abnormalities, has become a pressing problem in today’s large volume of Internet traffic. To this end, various anomaly detection techniques have been developed, including techniques based on long-range dependence (LRD) behavior estimation of network traffic. However, the existing LRD-based techniques analyze the aggregated WHOLE (control plus data) traffic, which might not be sufficient to detect short-duration and low-volume attacks and abnormalities in the traffic. This is because such anomalies might pass unnoticed in large volume of the normal background traffic. To address this issue, we propose a method that examines the LRD behavior of control and data planes traffic separately, which improves the detection efficacy. For LRD behavior analysis, the proposed method integrates the correlation structures of second-order self-similar and fractional autoregressive integrated moving average models. The performance of the proposed method is empirically evaluated and validated over a relatively recent real Internet traffic captured at King Saud University’s network. The analysis and results demonstrate that the proposed method efficiently detects such low volume and short duration attacks and abnormalities in the traffic, which would not be detected by merely analyzing the aggregated WHOLE traffic without decomposing it into control and data planes traffic.
---
paper_title: Wearable Sensor-Based Behavioral Anomaly Detection in Smart Assisted Living Systems

paper_content:

Detecting behavioral anomalies in human daily life is important to developing smart assisted-living systems for elderly care. Based on data collected from wearable motion sensors and the associated locational context, this paper presents a coherent anomaly detection framework to effectively detect different behavioral anomalies in human daily life. Four types of anomalies, including spatial anomaly, timing anomaly, duration anomaly, and sequence anomaly, are detected using a probabilistic theoretical framework. This framework is based on complex activity recognition using dynamic Bayesian network modeling. The maximum-likelihood estimation algorithm and Laplace smoothing are used in learning the parameters in the anomaly detection model. We conducted experimental evaluation in a mock apartment environment, and the results verified the effectiveness of the proposed framework. We expect that this behavioral anomaly detection system can be integrated into future smart homes for elderly care.
---
paper_title: Detection of Integrity Attacks in Cyber-Physical Critical Infrastructures Using Ensemble Modeling

paper_content:

This paper presents an anomaly-based methodology for reliable detection of integrity attacks in cyber-physical critical infrastructures. Such malicious events compromise the smooth operation of the infrastructure while the attacker is able to exploit the respective resources according to his/her purposes. Even though the operator may not understand the attack, since the overall system appears to remain in a steady state, the consequences may be of catastrophic nature with a huge negative impact. Here, we apply a computational intelligent technique which incorporates the merits of two of the heterogeneous modeling approaches (linear time-invariant and neural networks), while considering both temporal and functional dependencies existing among the elements of an infrastructure. The experimental platform includes a power grid simulator of the IEEE 30 bus model and a cyber network emulator. Subsequently, we implemented a wide range of integrity attacks (replay, ramp, pulse, scaling, and random) with different intensity levels. A thorough evaluation procedure is carried out while the results demonstrate the ability of the proposed method to produce a desired result in terms of false positive rate, false negative rate, and detection delay.
---
paper_title: Outlier Detection Techniques for Wireless Sensor Networks: A Survey

paper_content:

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.
---
paper_title: Outliers detection and classification in wireless sensor networks

paper_content:

Abstract In the past few years, many wireless sensor networks had been deployed in the real world to collect large amounts of raw sensed data. However, the key challenge is to extract high-level knowledge from such raw data. In the applications of sensor networks, outlier/anomaly detection has been paid more and more attention. Outlier detection can be used to filter noisy data, find faulty nodes, and discover interesting events. In this paper we propose a novel in-network knowledge discovery approach that provides outlier detection and data clustering simultaneously. Our approach is capable to distinguish between an error due to faulty sensor and an error due to an event (probably an environmental event) which characterize the spatial and temporal correlations between events observed by sensor nodes in a confined network neighborhood. Experiments on both synthetic and real datasets show that the proposed algorithm outperforms other techniques in both effectiveness and efficiency.
---
paper_title: Efficient Computer Network Anomaly Detection by Changepoint Detection Methods

paper_content:

We consider the problem of efficient on-line anomaly detection in computer network traffic. The problem is approached statistically, as that of sequential (quickest) changepoint detection. A multi-cyclic setting of quickest change detection is a natural fit for this problem. We propose a novel score-based multi-cyclic detection algorithm. The algorithm is based on the so-called Shiryaev-Roberts procedure. This procedure is as easy to employ in practice and as computationally inexpensive as the popular Cumulative Sum chart and the Exponentially Weighted Moving Average scheme. The likelihood ratio based Shiryaev-Roberts procedure has appealing optimality properties, particularly it is exactly optimal in a multi-cyclic setting geared to detect a change occurring at a far time horizon. It is therefore expected that an intrusion detection algorithm based on the Shiryaev-Roberts procedure will perform better than other detection schemes. This is confirmed experimentally for real traces. We also discuss the possibility of complementing our anomaly detection algorithm with a spectral-signature intrusion detection system with false alarm filtering and true attack confirmation capability, so as to obtain a synergistic system.
---
paper_title: Sliding Modes for Anomaly Observation in TCP Networks: From Theory to Practice

paper_content:

Anomaly detection has been an active open problem in the networks community for several years. In this brief, we aim at detecting such abnormal signals by control theory techniques. Several classes of sliding mode observers are proposed for a fluid flow model of the transmission control protocol (TCP)/internet protocol network. Comparative simulations via network simulator NS-2 show the enhancement brought by a higher order sliding mode observer. The efficiency of this observer opens the way toward observing traffics with real TCP flow characteristics. To achieve this end, trace replay techniques for TCP traffic traces are presented. Finally, experiments lead to successful anomaly estimation under real traffic conditions.
---
paper_title: Online and Scalable Unsupervised Network Anomaly Detection Method

paper_content:

Nowadays, network intrusion detectors mainly rely on knowledge databases to detect suspicious traffic. These databases have to be continuously updated which requires important human resources and time. Unsupervised network anomaly detectors overcome this issue by using “intelligent” techniques to identify anomalies without any prior knowledge. However, these systems are often very complex as they need to explore the network traffic to identify flows patterns. Therefore, they are often unable to meet real-time requirements. In this paper, we present a new online and real-time unsupervised network anomaly detection algorithm (ORUNADA). Our solution relies on a discrete time-sliding window to update continuously the feature space and an incremental grid clustering to detect rapidly the anomalies. The evaluations showed that ORUNADA can process online large network traffic while ensuring a low detection delay and good detection performance. The experiments performed on the traffic of a core network of a Spanish intermediate Internet service provider demonstrated that ORUNADA detects in less than half a second an anomaly after its occurrence. Furthermore, the results highlight that our solution outperforms in terms of true positive rate and false positive rate existing techniques reported in the literature.
---
paper_title: Scalable Anomaly Detection for Smart City Infrastructure Networks

paper_content:

Dynamically detecting anomalies can be difficult in very large-scale infrastructure networks. The authors' approach addresses spatiotemporal anomaly detection in a smarter city context with large numbers of sensors deployed. They propose a scalable, hybrid Internet infrastructure for dynamically detecting potential anomalies in real time using stream processing. The infrastructure enables analytically inspecting and comparing anomalies globally using large-scale array processing. Deployed on a real pipe network topology of 1,891 nodes, this approach can effectively detect and characterize anomalies while minimizing the amount of data shared across the network.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: Distributed anomaly detection for industrial wireless sensor networks based on fuzzy data modelling

paper_content:

Modern infrastructure increasingly depends on large computerized systems for their reliable operation. Supervisory Control and Data Acquisition (SCADA) systems are being deployed to monitor and control large scale distributed infrastructures (e.g. power plants, water distribution systems). A recent trend is to incorporate Wireless Sensor Networks (WSNs) to sense and gather data. However, due to the broadcast nature of the network and inherent limitations in the sensor nodes themselves, they are vulnerable to different types of security attacks. Given the critical aspects of the underlying infrastructure it is an extremely important research challenge to provide effective methods to detect malicious activities on these networks. This paper proposes a robust and scalable mechanism that aims to detect malicious anomalies accurately and efficiently using distributed in-network processing in a hierarchical framework. Unsupervised data partitioning is performed distributively adapting fuzzy c-means clustering in an incremental model. Non-parametric and non-probabilistic anomaly detection is performed through fuzzy membership evaluations and thresholds on observed inter-cluster distances. Robust thresholds are determined adaptively using second order statistical knowledge at each evaluation stage. Extensive experiments were performed and the results demonstrate that the proposed framework achieves high detection accuracy compared to existing data clustering approaches with more than 96% less communication overheads opposed to a centralized approach.
---
paper_title: A survey of network anomaly detection techniques

paper_content:

Information and Communication Technology (ICT) has a great impact on social wellbeing, economic growth and national security in todays world. Generally, ICT includes computers, mobile communication devices and networks. ICT is also embraced by a group of people with malicious intent, also known as network intruders, cyber criminals, etc. Confronting these detrimental cyber activities is one of the international priorities and important research area. Anomaly detection is an important data analysis task which is useful for identifying the network intrusions. This paper presents an in-depth analysis of four major categories of anomaly detection techniques which include classification, statistical, information theory and clustering. The paper also discusses research challenges with the datasets used for network intrusion detection. HighlightsMaps different types of anomalies with network attacks.Provides an up-to-date taxonomy of network anomaly detection.Evaluates effectiveness of different categories of techniques.Explores recent research related to publicly available network intrusion evaluation datasets.
---
paper_title: Network Anomaly Detection: Methods, Systems and Tools

paper_content:

Network anomaly detection is an important and dynamic research area. Many network intrusion detection methods and systems (NIDS) have been proposed in the literature. In this paper, we provide a structured and comprehensive overview of various facets of network anomaly detection so that a researcher can become quickly familiar with every aspect of network anomaly detection. We present attacks normally encountered by network intrusion detection systems. We categorize existing network anomaly detection methods and systems based on the underlying computational techniques used. Within this framework, we briefly describe and compare a large number of network anomaly detection methods and systems. In addition, we also discuss tools that can be used by network defenders and datasets that researchers in network anomaly detection can use. We also highlight research directions in network anomaly detection.
---
paper_title: Wireless Anomaly Detection Based on IEEE 802.11 Behavior Analysis

paper_content:

Wireless communication networks are pervading every aspect of our lives due to their fast, easy, and inexpensive deployment. They are becoming ubiquitous and have been widely used to transfer critical information, such as banking accounts, credit cards, e-mails, and social network credentials. The more pervasive the wireless technology is going to be, the more important its security issue will be. Whereas the current security protocols for wireless networks have addressed the privacy and confidentiality issues, there are unaddressed vulnerabilities threatening their availability and integrity (e.g., denial of service, session hijacking, and MAC address spoofing attacks). In this paper, we describe an anomaly based intrusion detection system for the IEEE 802.11 wireless networks based on behavioral analysis to detect deviations from normal behaviors that are triggered by wireless network attacks. Our anomaly behavior analysis of the 802.11 protocols is based on monitoring the n-consecutive transitions of the protocol state machine. We apply sequential machine learning techniques to model the n-transition patterns in the protocol and characterize the probabilities of these transitions being normal. We have implemented several experiments to evaluate our system performance. By cross validating the system over two different wireless channels, we have achieved a low false alarm rate (<0.1%). We have also evaluated our approach against an attack library of known wireless attacks and has achieved more than 99% detection rate.
---
paper_title: Outside the Closed World: On Using Machine Learning for Network Intrusion Detection

paper_content:

In network intrusion detection research, one popular strategy for finding attacks is monitoring a network's activity for anomalies: deviations from profiles of normality previously learned from benign traffic, typically identified using tools borrowed from the machine learning community. However, despite extensive academic research one finds a striking gap in terms of actual deployments of such systems: compared with other intrusion detection approaches, machine learning is rarely employed in operational "real world" settings. We examine the differences between the network intrusion detection problem and other areas where machine learning regularly finds much more success. Our main claim is that the task of finding attacks is fundamentally different from these other applications, making it significantly harder for the intrusion detection community to employ machine learning effectively. We support this claim by identifying challenges particular to network intrusion detection, and provide a set of guidelines meant to strengthen future research on anomaly detection.
---
paper_title: An Efficient Network Log Anomaly Detection System Using Random Projection Dimensionality Reduction

paper_content:

Network traffic is increasing all the time and network services are becoming more complex and vulnerable. To protect these networks, intrusion detection systems are used. Signature-based intrusion detection cannot find previously unknown attacks, which is why anomaly detection is needed. However, many new systems are slow and complicated. We propose a log anomaly detection framework which aims to facilitate quick anomaly detection and also provide visualizations of the network traffic structure. The system preprocesses network logs into a numerical data matrix, reduces the dimensionality of this matrix using random projection and uses Mahalanobis distance to find outliers and calculate an anomaly score for each data point. Log lines that are too different are flagged as anomalies. The system is tested with real-world network data, and actual intrusion attempts are found. In addition, visualizations are created to represent the structure of the network data. We also perform computational time evaluation to ensure the performance is feasible. The system is fast, finds intrusion attempts and does not need clean training data.
---
paper_title: Network anomaly detection using IP flows with Principal Component Analysis and Ant Colony Optimization

paper_content:

It is remarkable how proactive network management is in such demand nowadays, since networks are growing in size and complexity and Information Technology services cannot be stopped. In this manner, it is necessary to use an approach which proactively identifies traffic behavior patterns which may harm the network's normal operations. Aiming an automated management to detect and prevent potential problems, we present and compare two novel anomaly detection mechanisms based on statistical procedure Principal Component Analysis and the Ant Colony Optimization metaheuristic. These methods generate a traffic profile, called Digital Signature of Network Segment using Flow analysis (DSNSF), which is adopted as normal network behavior. Then, this signature is compared with the real network traffic by using a modification of the Dynamic Time Warping metric in order to recognize anomalous events. Thus, a seven-dimensional analysis of IP flows is performed, allowing the characterization of bits, packets and flows traffic transmitted per second, and the extraction of descriptive flow attributes, like source IP address, destination IP address, source TCP/UDP port and destination TCP/UDP port. The systems were evaluated using a real network environment and showed promising results. Moreover, the correspondence between true-positive and false-positive rates demonstrates that the systems are able to enhance the detection of anomalous behavior by maintaining a satisfactory false-alarm rate. Display Omitted Anomaly detection issue is addressed based on network traffic profiling.Proposal and comparison of detection methods belonging to distinct algorithm classes.Detection mechanism constructed over an adaptation of a pattern matching technique.Use of real and simulated traffic to evaluate the proposed methods.Traffic patterns that may harm the network operations are proactively identified.
---
paper_title: Ensembles of incremental learners to detect anomalies in ad hoc sensor networks

paper_content:

Display Omitted In the past decade, rapid technological advances in the fields of electronics and telecommunications have given rise to versatile, ubiquitous decentralized embedded sensor systems with ad hoc wireless networking capabilities. Typically these systems are used to gather large amounts of data, while the detection of anomalies (such as system failures, intrusion, or unanticipated behavior of the environment) in the data (or other types or processing) is performed in centralized computer systems. In spite of the great interest that it attracts, the systematic porting and analysis of centralized anomaly detection algorithms to a decentralized paradigm (compatible with the aforementioned sensor systems) has not been thoroughly addressed in the literature. We approach this task from a new angle, assessing the viability of localized (in-node) anomaly detection based on machine learning. The main challenges we address are: (1) deploying decentralized, automated, online learning, anomaly detection algorithms within the stringent constraints of typical embedded systems; and (2) evaluating the performance of such algorithms and comparing them with that of centralized ones. To this end, we first analyze (and port) single and multi-dimensional input classifiers that are trained incrementally online and whose computational requirements are compatible with the limitations of embedded platforms. Next, we combine multiple classifiers in a single online ensemble. Then, using both synthetic and real-world datasets from different application domains, we extensively evaluate the anomaly detection performance of our algorithms and ensemble, in terms of precision and recall, and compare it to that of well-known offline, centralized machine learning algorithms. Our results show that the ensemble performs better than each individual decentralized classifier and that it can match the performance of the offline alternatives, thus showing that our approach is a viable solution to detect anomalies, even in environments with little a priori knowledge.
---
paper_title: Outlier detection: A survey on techniques of WSNs involving event and error based outliers

paper_content:

In the recent few years, many wireless sensor networks have been distributed systematically in the real world to collect valuable raw sensed data. However, the crucial point of challenge is to extract high level knowledge from this raw sensed data. In the application of data analysis, a necessary preprocessing step is anomaly detection, also known as deviation detection or data cleansing. Outliers in wireless sensor networks (WSNs) are those measures that deviate from a defined pattern. Outlier detection can be used to remove noisy data, detect faulty nodes and discover interesting events. Numerous small and low cost nodes loaded with capabilities of integrated sensing and computation are involved in a WSN structure. Due to high density WSNs are exposed to faults and nasty attacks causing inaccurate and unreliable sensors reading, making Wireless sensor networks prone to outliers. This survey provides an outline of outlier detection techniques and approaches focusing on event and error based outliers.
---
paper_title: Learning combination of anomaly detectors for security domain

paper_content:

This paper presents a novel technique of finding a convex combination of outputs of anomaly detectors maximizing the accuracy in ź-quantile of most anomalous samples. Such an approach better reflects the needs in the security domain in which subsequent analysis of alarms is costly and can be done only on a small number of alarms. An extensive experimental evaluation and comparison to prior art on real network data using sets of anomaly detectors of two existing intrusion detection systems shows that the proposed method not only outperforms prior art, it is also more robust to noise in training data labels, which is another important feature for deployment in practice.
---
paper_title: Analysis of HTTP Requests for Anomaly Detection of Web Attacks

paper_content:

Attacks against web servers and web-based applications remain a serious global network security threat. Attackers are able to compromise web services, collect confidential information from web data bases, interrupt or completely paralyze web servers. In this study, we consider the analysis of HTTP logs for the detection of network intrusions. First, a training set of HTTP requests which does not contain any attacks is analyzed. When all relevant information has been extracted from the logs, several clustering and anomaly detection algorithms are employed to describe the model of normal users behavior. This model is then used to detect network attacks as deviations from the norms in an online mode. The simulation results presented show that, compared to other data mining algorithms, the method results in a higher accuracy rate.
---
paper_title: Network Traffic Classification Using Correlation Information

paper_content:

Traffic classification has wide applications in network management, from security monitoring to quality of service measurements. Recent research tends to apply machine learning techniques to flow statistical feature based classification methods. The nearest neighbor (NN)-based method has exhibited superior classification performance. It also has several important advantages, such as no requirements of training procedure, no risk of overfitting of parameters, and naturally being able to handle a huge number of classes. However, the performance of NN classifier can be severely affected if the size of training data is small. In this paper, we propose a novel nonparametric approach for traffic classification, which can improve the classification performance effectively by incorporating correlated information into the classification process. We analyze the new classification approach and its performance benefit from both theoretical and empirical perspectives. A large number of experiments are carried out on two real-world traffic data sets to validate the proposed approach. The results show the traffic classification performance can be improved significantly even under the extreme difficult circumstance of very few training samples.
---
paper_title: Wearable Sensor-Based Behavioral Anomaly Detection in Smart Assisted Living Systems

paper_content:

Detecting behavioral anomalies in human daily life is important to developing smart assisted-living systems for elderly care. Based on data collected from wearable motion sensors and the associated locational context, this paper presents a coherent anomaly detection framework to effectively detect different behavioral anomalies in human daily life. Four types of anomalies, including spatial anomaly, timing anomaly, duration anomaly, and sequence anomaly, are detected using a probabilistic theoretical framework. This framework is based on complex activity recognition using dynamic Bayesian network modeling. The maximum-likelihood estimation algorithm and Laplace smoothing are used in learning the parameters in the anomaly detection model. We conducted experimental evaluation in a mock apartment environment, and the results verified the effectiveness of the proposed framework. We expect that this behavioral anomaly detection system can be integrated into future smart homes for elderly care.
---
paper_title: A Survey of Outlier Detection Methods in Network Anomaly Identification

paper_content:

The detection of outliers has gained considerable interest in data mining with the realization that outliers can be the key discovery to be made from very large databases. Outliers arise due to various reasons such as mechanical faults, changes in system behavior, fraudulent behavior, human error and instrument error. Indeed, for many applications the discovery of outliers leads to more interesting and useful results than the discovery of inliers. Detection of outliers can lead to identification of system faults so that administrators can take preventive measures before they escalate. It is possible that anomaly detection may enable detection of new attacks. Outlier detection is an important anomaly detection approach. In this paper, we present a comprehensive survey of well-known distance-based, density-based and other techniques for outlier detection and compare them. We provide definitions of outliers and discuss their detection based on supervised and unsupervised learning in the context of network anomaly detection.
---
paper_title: Bayesian Mechanisms and Detection Methods for Wireless Network with Malicious Users

paper_content:

Strategic users in a wireless network cannot be assumed to follow the network algorithms blindly. Moreover, some of these users aim to use their knowledge about network algorithms to maliciously gain more resources and also to create interference to other users. We consider a scenario, in which the network and legitimate users gather probabilistic information about the presence of malicious users by observing the network over a long time period. The network (mechanism designer) and legitimate users modify their actions according to this Bayesian information. We consider Bayesian mechanisms, both pricing schemes and auctions, and obtain the Bayesian Nash Equilibrium (BNE) points. The BNE points provide conditions under which, the uncertainty about user's nature (type) is better for regular (legitimate) users. To derive these conditions, we compare the Bayesian case to the complete information case. We obtain the optimal prices and allocations, which counter the malicious users. We also provide detection methods based on machine learning algorithms for the detection of malicious users, by observing the prices and rate allocations. In addition, we provide detection using regression learning by observing the anomalies in the utility functions of malicious users from prices, which is implemented along with the pricing mechanism itself. For the designer and the regular users, in a complementary fashion, the results of the detections provide a better estimate of the statistics of malicious users to implement the pricing mechanisms. We have also proposed a truthful Bayesian mechanism in the presence of malicious users. The numerical studies for malicious user detection are carried out with the model proposed in the paper as well as using real Botnet dataset.
---
paper_title: High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

paper_content:

High-dimensional problem domains pose significant challenges for anomaly detection. The presence of irrelevant features can conceal the presence of anomalies. This problem, known as the 'curse of dimensionality', is an obstacle for many anomaly detection techniques. Building a robust anomaly detection model for use in high-dimensional spaces requires the combination of an unsupervised feature extractor and an anomaly detector. While one-class support vector machines are effective at producing decision surfaces from well-behaved feature vectors, they can be inefficient at modelling the variation in large, high-dimensional datasets. Architectures such as deep belief networks (DBNs) are a promising technique for learning robust features. We present a hybrid model where an unsupervised DBN is trained to extract generic underlying features, and a one-class SVM is trained from the features learned by the DBN. Since a linear kernel can be substituted for nonlinear ones in our hybrid model without loss of accuracy, our model is scalable and computationally efficient. The experimental results show that our proposed model yields comparable anomaly detection performance with a deep autoencoder, while reducing its training and testing time by a factor of 3 and 1000, respectively. HighlightsWe use a combination of a one-class SVM and deep learning.In our model linear kernels can be used rather than nonlinear ones.Our model delivers a comparable accuracy with a deep autoencoder.Our model executes 3times faster in training and 1000 faster than a deep autoencoder.
---
paper_title: Network Anomaly Detection System using Genetic Algorithm and Fuzzy Logic

paper_content:

Multiple attributes from IP flows are combined to detect anomalous events.GA metaheuristic used for Digital Signature of Network Segment using Flow Analysis.Unsupervised training technique applied efficiently for network traffic profiling.Fuzzy Logic improved accuracy and false positives compared to state of art. Due to the sheer number of applications that uses computer networks, in which some are crucial to users and enterprises, network management is essential. Therefore, integrity and availability of computer networks become priorities, making it a fundamental resource to be managed. In this work, a scheme combining Genetic Algorithm and a Fuzzy Logic for network anomaly detection is discussed. The Genetic Algorithm is used to generate a Digital Signature of Network Segment using Flow Analysis, where information extracted from network flows data is used to predict the networks traffic behavior for a given time interval. Furthermore, a Fuzzy Logic scheme is applied to decide whether an instance represents an anomaly or not, differing from some approaches present in the literature. Indeed, it is proposed an expert system with the capability to monitor the networks traffic with IP flows while expected behaviors are generated in a regular time interval basis, issuing alarms when a possible problem is present. The proposed anomaly detection system exposes network problems autonomously. The results acquired from applying the proposed approach in a real network traffic flows achieve an accuracy of 96.53% and false positive rate of 0.56%. Moreover, our method succeeds in achieving higher performance compared to several other approaches.
---
paper_title: Outlier Detection Techniques for Wireless Sensor Networks: A Survey

paper_content:

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.
---
paper_title: Outliers detection and classification in wireless sensor networks

paper_content:

Abstract In the past few years, many wireless sensor networks had been deployed in the real world to collect large amounts of raw sensed data. However, the key challenge is to extract high-level knowledge from such raw data. In the applications of sensor networks, outlier/anomaly detection has been paid more and more attention. Outlier detection can be used to filter noisy data, find faulty nodes, and discover interesting events. In this paper we propose a novel in-network knowledge discovery approach that provides outlier detection and data clustering simultaneously. Our approach is capable to distinguish between an error due to faulty sensor and an error due to an event (probably an environmental event) which characterize the spatial and temporal correlations between events observed by sensor nodes in a confined network neighborhood. Experiments on both synthetic and real datasets show that the proposed algorithm outperforms other techniques in both effectiveness and efficiency.
---
paper_title: Mobile Payment Anomaly Detection Mechanism Based on Information Entropy

paper_content:

With the rapid growth in the number of mobile phone users, mobile payments have become an important part of mobile e-commerce applications. Secure payment systems directly affect the security of e-commerce systems. This study proposes an anomaly detection mechanism supported by an information entropy method combined with neural network to improve mobile payments security. As the entropy value is sensitive and have much difference between normal and abnormal traffic flow in the mobile payment system, the abnormal traffic data will be detected. The simulation result shows that it can realise the effective monitoring of abnormal flow in the mobile payment system.
---
paper_title: On the symbiosis of specification-based and anomaly-based detection

paper_content:

As the number of attacks on computer systems increases and become more sophisticated, there is an obvious need for intrusion detection systems to be able to effectively recognize the known attacks and adapt to novel threats. The specification-based intrusion detection has been long considered as a promising solution that integrates the characteristics of ideal intrusion detection system: the accuracy of detection and ability to recognize novel attacks. However, one of the main challenges of applying this technique in practice is its dependence on the user guidance in developing the specification of normal system behavior. In this work, we present an approach for automatic generation of specifications for any software systems executing on a single host based on the combination of two techniques: specification-based and anomaly-based approaches. The proposed technique allows automatic development of the normal and abnormal behavioral specifications in a form of variable-length patterns classified via anomaly-based approach. Specifically, we use machine-learning algorithm to classify fixed-length patterns generated via sliding window technique to infer the classification of variable-length patterns from the aggregation of the machine learning based classification results. We describe the design and implementation of our technique and show its practical applicability in the domain of security monitoring through simulation and experiments.
---
paper_title: Semi-supervised Statistical Approach for Network Anomaly Detection

paper_content:

Abstract Intrusion Detection Systems (IDS) have become a very important defense measure against security threats. In recent years, computer networks are widely deployed for critical and complex systems, which make them more vulnerable to network attacks. In this paper, we propose a two-stage Semi-supervised Statistical approach for Anomaly Detection (SSAD). The first stage of SSAD aims to build a probabilistic model of normal instances and measures any deviation that exceeds an established threshold. This threshold is deduced from a regularized discriminant function of Maximum Likelihood (ML). The purpose of the second stage is to reduce False Alarm Rate (FAR) through an iterative process that reclassifies anomaly cluster, from the first stage, using a similarity distance and anomaly's cluster dispersion rate. We evaluate the proposed approach on the well-known intrusion detection dataset NSL-KDD and Kyoto 2006+. The experimental results show that SSAD outperforms the Naive Bayes methods in terms of Detection Rate and False Positive Rate.
---
paper_title: Hyperspherical cluster based distributed anomaly detection in wireless sensor networks

paper_content:

This article describes a distributed hyperspherical cluster based algorithm for identifying anomalies in measurements from a wireless sensor network, and an implementation on a real wireless sensor network testbed. The communication overhead incurred in the network is minimised by clustering sensor measurements and merging clusters before sending a compact description of the clusters to other nodes. An evaluation on several real and synthetic datasets demonstrates that the distributed hyperspherical cluster-based scheme achieves comparable detection accuracy with a significant reduction in communication overhead compared to a centralised scheme, where all the sensor node measurements are communicated to a central node for processing. Detecting anomalies in data is challenging on resource constrained networks.We propose a distributed algorithm using hyperspherical cluster based data models.The scheme is capable of identifying global anomalies at an individual node level.Comparable detection accuracy with significant reduction in communication overhead.Implemented and demonstrated on a real wireless sensor network test-bed.
---
paper_title: Distributed Anomaly Detection Using Minimum Volume Elliptical Principal Component Analysis

paper_content:

Principal component analysis and the residual error is an effective anomaly detection technique. In an environment where anomalies are present in the training set, the derived principal components can be skewed by the anomalies. A further aspect of anomaly detection is that data might be distributed across different nodes in a network and their communication to a centralized processing unit is prohibited due to communication cost. Current solutions to distributed anomaly detection rely on a hierarchical network infrastructure to aggregate data or models; however, in this environment, links close to the root of the tree become critical and congested. In this paper, an algorithm is proposed that is more robust in its derivation of the principal components of a training set containing anomalies. A distributed form of the algorithm is then derived where each node in a network can iterate towards the centralized solution by exchanging small matrices with neighboring nodes. Experimental evaluations on both synthetic and real-world data sets demonstrate the superior performance of the proposed approach in comparison to principal component analysis and alternative anomaly detection techniques. In addition, it is shown that in a variety of network infrastructures, the distributed form of the anomaly detection model is able to derive a close approximation of the centralized model.
---
paper_title: ENTROPY AND FLOW-BASED APPROACH FOR ANOMALOUS TRAFFIC FILTERING

paper_content:

Research Tools in Anomaly-based Intrusion Detection are highly dependent on appropriate traffic trace data. Traditional datasets present several issues such as: removal of sensitive information (anonymization) and insufficient number or volume of attack instances, which limit their quality for the design and evaluation of A-NIDSs. In this paper, we present a method for anomalous traffic filtering which can be used for generating anomaly-free traffic traces. The sanitized dataset can be used to improve the computation of the behaviour profiles during the training stage. The proposal is based on the construction and statistical analysis of the flow-level entropy space for the identification of outliers using three entropy estimators. Empirical results showed that the new traffic traces of the sanitized dataset have a distributional similarity among them greater than that presented among the original datasets.
---
paper_title: A Family of Joint Sparse PCA Algorithms for Anomaly Localization in Network Data Streams

paper_content:

Determining anomalies in data streams that are collected and transformed from various types of networks has recently attracted significant research interest. Principal component analysis (PCA) has been extensively applied to detecting anomalies in network data streams. However, none of existing PCA-based approaches addresses the problem of identifying the sources that contribute most to the observed anomaly, or anomaly localization. In this paper, we propose novel sparse PCA methods to perform anomaly detection and localization for network data streams. Our key observation is that we can localize anomalies by identifying a sparse low-dimensional space that captures the abnormal events in data streams. To better capture the sources of anomalies, we incorporate the structure information of the network stream data in our anomaly localization framework. Furthermore, we extend our joint sparse PCA framework with multidimensional Karhunen Loeve Expansion that considers both spatial and temporal domains of data streams to stabilize localization performance. We have performed comprehensive experimental studies of the proposed methods and have compared our methods with the state-of-the-art using three real-world data sets from different application domains. Our experimental studies demonstrate the utility of the proposed methods.
---
paper_title: Detecting anomalies from big network traffic data using an adaptive detection approach

paper_content:

The unprecedented explosion of real-life big data sets have sparked a lot of research interests in data mining in recent years. Many of these big data sets are generated in network environment and are characterized by a dauntingly large size and high dimensionality which pose great challenges for detecting useful knowledge and patterns, such as network traffic anomalies, from them. In this paper, we study the problem of anomaly detection in big network connection data sets and propose an outlier detection technique, called Adaptive Stream Projected Outlier deTector (A-SPOT), to detect anomalies from large data sets using a novel adaptive subspace analysis approach. A case study of A-SPOT is conducted in this paper by deploying it to the 1999 KDD CUP anomaly detection application. Innovative approaches for training data generation, anomaly classification and false positive reduction are proposed in this paper as well to better tailor A-SPOT to deal with the case study. Experimental results demonstrate that A-SPOT is effective and efficient in detecting anomalies from network data sets and outperforms existing detection methods.
---
paper_title: A Survey of Distance and Similarity Measures Used Within Network Intrusion Anomaly Detection

paper_content:

Anomaly detection (AD) use within the network intrusion detection field of research, or network intrusion AD (NIAD), is dependent on the proper use of similarity and distance measures, but the measures used are often not documented in published research. As a result, while the body of NIAD research has grown extensively, knowledge of the utility of similarity and distance measures within the field has not grown correspondingly. NIAD research covers a myriad of domains and employs a diverse array of techniques from simple $k$ -means clustering through advanced multiagent distributed AD systems. This review presents an overview of the use of similarity and distance measures within NIAD research. The analysis provides a theoretical background in distance measures and a discussion of various types of distance measures and their uses. Exemplary uses of distance measures in published research are presented, as is the overall state of the distance measure rigor in the field. Finally, areas that require further focus on improving the distance measure rigor in the NIAD field are presented.
---
paper_title: Systematic literature reviews in software engineering – A systematic literature review

paper_content:

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.
---
paper_title: Big Data Analytics for User-Activity Analysis and User-Anomaly Detection in Mobile Wireless Network

paper_content:

The next generation wireless networks are expected to operate in fully automated fashion to meet the burgeoning capacity demand and to serve users with superior quality of experience. Mobile wireless networks can leverage spatio-temporal information about user and network condition to embed the system with end-to-end visibility and intelligence. Big data analytics has emerged as a promising approach to unearth meaningful insights and to build artificially intelligent models with assistance of machine learning tools. Utilizing aforementioned tools and techniques, this paper contributes in two ways. First, we utilize mobile network data (Big Data)—call detail record—to analyze anomalous behavior of mobile wireless network. For anomaly detection purposes, we use unsupervised clustering techniques namely k-means clustering and hierarchical clustering. We compare the detected anomalies with ground truth information to verify their correctness. From the comparative analysis, we observe that when the network experiences abruptly high (unusual) traffic demand at any location and time, it identifies that as anomaly. This helps in identifying regions of interest in the network for special action such as resource allocation, fault avoidance solution, etc. Second, we train a neural-network-based prediction model with anomalous and anomaly-free data to highlight the effect of anomalies in data while training/building intelligent models. In this phase, we transform our anomalous data to anomaly-free and we observe that the error in prediction, while training the model with anomaly-free data has largely decreased as compared to the case when the model was trained with anomalous data.
---
paper_title: PCA-Based Network Traffic Anomaly Detection

paper_content:

The use of a Traffic Matrix (TM) to describe the characteristics of a global network has attracted significant interest in network performance research. Due to the high dimensionality and sparsity of network traffic, Principal Component Analysis (PCA) has been successfully applied to TM analysis. PCA is one of the most common methods used in analysis of high-dimensional objects. This paper shows how to apply PCA to TM analysis and anomaly detection. The experiment results demonstrate that the PCA-based method can detect anomalies for both single and multiple nodes with high accuracy and efficiency.
---
paper_title: An anomalous behavior detection model in cloud computing

paper_content:

This paper proposes an anomalous behavior detection model based on cloud computing. Virtual Machines (VMs) are one of the key components of cloud Infrastructure as a Service (IaaS). The security of such VMs is critical to IaaS security. Many studies have been done on cloud computing security issues, but research into VM security issues, especially regarding VM network traffic anomalous behavior detection, remains inadequate. More and more studies show that communication among internal nodes exhibits complex patterns. Communication among VMs in cloud computing is invisible. Researchers find such issues challenging, and few solutions have been proposed-leaving cloud computing vulnerable to network attacks. This paper proposes a model that uses Software-Defined Networks (SDN) to implement traffic redirection. Our model can capture inter-VM traffic, detect known and unknown anomalous network behaviors, adopt hybrid techniques to analyze VM network behaviors, and control network systems. The experimental results indicate that the effectiveness of our approach is greater than 90%, and prove the feasibility of the model.
---
paper_title: Effective Statistical Detection of Smart Confidentiality Attacks in Multi-Domain Networks

paper_content:

The need to preserve information confidentiality among network providers has prevented the actual deployment of effective Traffic Engineering (TE) solutions for QoS-enabled end-to-end connectivity services in multi-domain multi-provider networks. The use of Path Computation Element (PCE) architecture can foster the effective implementation of TE through a centralized engine devoted to end-to-end path computations. However, despite authentication, authorization and encryption, confidentiality issues may arise due to abuses of information included in path computation replies to bogus requests issued by malicious PCEs. This paper first demonstrates the security leak allowing the exposure of intra-domain information in current inter-PCE path computation procedures. Then it proposes an anomaly-based approach, namely PCE Anomaly Detector (PAD) in order to detect malicious utilization of path computation services. The proposed PAD employs a novel double-step multi-dimensional formulation based on the Sequential Hypothesis Testing (SHT) statistical classification procedure, able to recognize a suspicious sequence of requests while aiming at inferring confidential information about other domains. PAD is extensively evaluated through simulations. Results show good performance in terms of detection capabilities while guaranteeing the trade-off between accuracy and responsiveness, minimizing false alarm occurrences. Robustness against smart attacks is also proved with respect to a comprehensive set of attack patterns and under different network load conditions. Finally, intra-domain information exposition is evaluated, showing the PAD ability to preserve confidentiality.
---
paper_title: Statistical Traffic Anomaly Detection in Time-Varying Communication Networks

paper_content:

We propose two methods for traffic anomaly detection in communication networks where properties of normal traffic evolve dynamically. We formulate the anomaly detection problem as a binary composite hypothesis testing problem and develop a model-free and a model-based method, leveraging techniques from the theory of large deviations. Both methods first extract a family of probability laws (PLs) that represent normal traffic patterns during different time-periods, and then detect anomalies by assessing deviations of traffic from these laws. We establish the asymptotic Newman-Pearson optimality of both methods and develop an optimization-based approach for selecting the family of PLs from past traffic data. We validate our methods on networks with two representative time-varying traffic patterns and one common anomaly related to data exfiltration. Simulation results show that our methods perform better than their vanilla counterparts, which assume that normal traffic is stationary.
---
paper_title: Granular Evaluation of Anomalies in Wireless Sensor Networks Using Dynamic Data Partitioning with an Entropy Criteria

paper_content:

This paper presents an anomaly detection model that is granular and distributed to accurately and efficiently identify sensed data anomalies within wireless sensor networks. A more decentralised mechanism is introduced with wider use of in-network processing on a hierarchical sensor node topology resulting in a robust framework for dynamic data domains. This efficiently addresses the big data issue that is encountered in large scale industrial sensor network applications. Data vectors on each node’s observation domain is first partitioned using an unsupervised approach that is adaptive regarding dynamic data streams using cumulative point-wise entropy and average relative density . Second order statistical analysis applied on average relative densities and mean entropy values is then used to differentiate anomalies through robust and adaptive thresholds that are responsive to a dynamic environment. Anomaly detection is then performed in a non-parametric and non-probabilistic manner over the different network tiers in the hierarchical topology in offering increased granularity for evaluation. Experiments were performed extensively using both real and artificial data distributions representative of different dynamic and multi-density observation domains. Results demonstrate higher accuracies in detection as more than 94 percent accompanied by a desirable reduction of more than 85 percent in communication costs when compared to existing centralized methods.
---
paper_title: Segment-Based Anomaly Detection with Approximated Sample Covariance Matrix in Wireless Sensor Networks

paper_content:

In wireless sensor networks (WSNs), it has been observed that most abnormal events persist over a considerable period of time instead of being transient. As existing anomaly detection techniques usually operate in a point-based manner that handles each observation individually, they are unable to reliably and efficiently report such long-term anomalies appeared in an individual sensor node. Therefore, in this paper, we focus on a new technique for handling data in a segment-based manner. Considering a collection of neighbouring data segments as random variables, we determine those behaving abnormally by exploiting their spatial predictabilities and, motivated by spatial analysis, specifically investigate how to implement a prediction variance detector in a WSN. As the communication cost incurred in aggregating a covariance matrix is finally optimised using the Spearman’s rank correlation coefficient and differential compression, the proposed scheme is able to efficiently detect a wide range of long-term anomalies. In theory, comparing to the regular centralised approach, it can reduce the communication cost by approximately 80 percent. Moreover, its effectiveness is demonstrated by the numerical experiments, with a real world data set collected by the Intel Berkeley ResearchLab (IBRL).
---
paper_title: Detecting Insider Threats Using RADISH: A System for Real-Time Anomaly Detection in Heterogeneous Data Streams

paper_content:

We present a scalable system for high-throughput real-time analysis of heterogeneous data streams. Our architecture enables incremental development of models for predictive analytics and anomaly detection as data arrives into the system. In contrast with batch data-processing systems, such as Hadoop, that can have high latency, our architecture allows for ingest and analysis of data on the fly, thereby detecting and responding to anomalous behavior in near real time. This timeliness is important for applications such as insider threat, financial fraud, and network intrusions. We demonstrate an application of this system to the problem of detecting insider threats, namely, the misuse of an organization's resources by users of the system and present results of our experiments on a publicly available insider threat dataset.
---
paper_title: Network Volume Anomaly Detection and Identification in Large-scale Networks based on Online Time-structured Traffic Tensor Tracking

paper_content:

This paper addresses network anomography, that is, the problem of inferring network-level anomalies from indirect link measurements. This problem is cast as a low-rank subspace tracking problem for normal flows under incomplete observations and an outlier detection problem for abnormal flows. Since traffic data is large-scale time-structured data accompanied with noise and outliers under partial observations, an efficient modeling method is essential. To this end, this paper proposes an online subspace tracking of a Hankelized time-structured traffic tensor for normal flows based on the Candecomp/PARAFAC decomposition exploiting the recursive least squares algorithm. We estimate abnormal flows as outlier sparse flows via sparsity maximization in the underlying under-constrained linear-inverse problem. A major advantage is that our algorithm estimates normal flows by low-dimensional matrices with time-directional features as well as the spatial correlation of multiple links without using the past observed measurements and the past model parameters. Extensive numerical evaluations show that the proposed algorithm achieves faster convergence per iteration of model approximation and better volume anomaly detection performance compared to state-of-the-art algorithms.
---
paper_title: Mobile Payment Anomaly Detection Mechanism Based on Information Entropy

paper_content:

With the rapid growth in the number of mobile phone users, mobile payments have become an important part of mobile e-commerce applications. Secure payment systems directly affect the security of e-commerce systems. This study proposes an anomaly detection mechanism supported by an information entropy method combined with neural network to improve mobile payments security. As the entropy value is sensitive and have much difference between normal and abnormal traffic flow in the mobile payment system, the abnormal traffic data will be detected. The simulation result shows that it can realise the effective monitoring of abnormal flow in the mobile payment system.
---
paper_title: The BGP Visibility Toolkit: Detecting Anomalous Internet Routing Behavior

paper_content:

In this paper, we propose the BGP Visibility Toolkit, a system for detecting and analyzing anomalous behavior in the Internet. We show that interdomain prefix visibility can be used to single out cases of erroneous demeanors resulting from misconfiguration or bogus routing policies. The implementation of routing policies with BGP is a complicated process, involving fine-tuning operations and interactions with the policies of the other active ASes. Network operators might end up with faulty configurations or unintended routing policies that prevent the success of their strategies and impact their revenues. As part of the Visibility Toolkit, we propose the BGP Visibility Scanner, a tool which identifies limited visibility prefixes in the Internet. The tool enables operators to provide feedback on the expected visibility status of prefixes. We build a unique set of ground-truth prefixes qualified by their ASes as intended or unintended to have limited visibility. Using a machine learning algorithm, we train on this unique dataset an alarm system that separates with 95% accuracy the prefixes with unintended limited visibility. Hence, we find that visibility features are generally powerful to detect prefixes which are suffering from inadvertent effects of routing policies. Limited visibility could render a whole prefix globally unreachable. This points towards a serious problem, as limited reachability of a non-negligible set of prefixes undermines the global connectivity of the Internet. We thus verify the correlation between global visibility and global connectivity of prefixes.
---
paper_title: Distributed Real-Time Anomaly Detection in Networked Industrial Sensing Systems

paper_content:

Reliable real-time sensing plays a vital role in ensuring the reliability and safety of industrial cyberphysical systems (CPSs) such as wireless sensor and actuator networks. For many reasons, such as harsh industrial environments, fault-prone sensors, or malicious attacks, sensor readings may be abnormal or faulty. This could lead to serious system performance degradation or even catastrophic failure. Current anomaly detection approaches are either centralized and complicated or restricted due to strict assumptions, which are not suitable for practical large-scale networked industrial sensing systems (NISSs), where sensing devices are connected via digital communications, such as wireless sensor networks or smart grid systems. In this paper, we introduce a fully distributed general anomaly detection (GAD) scheme, which uses graph theory and exploits spatiotemporal correlations of physical processes to carry out real-time anomaly detection for general large-scale NISSs. We formally prove the scalability of our GAD approach and evaluate the performance of GAD for two industrial applications: building structure monitoring and smart grids. Extensive trace-driven simulations validate our theoretical analysis and demonstrate that our approach can significantly outperform state-of-the-art approaches in terms of detection accuracy and efficiency.
---
paper_title: Structure Regularized Traffic Monitoring for Traffic Matrix Estimation and Anomaly Detection by Link-Load Measurements

paper_content:

We consider problems of traffic matrix (TM) estimation and anomaly detection utilizing link-load traffic measurements. Models including structure regularized traffic monitoring (SRTM) and dynamic SRTM (DSRTM) are presented to realize traffic monitoring under static and dynamic routing configurations, respectively. Considering that real traffic data are usually approximately low-rank but exhibit strong spatial and temporal dependencies, we define spatial and temporal regularization matrices based on Moore–Penrose pseudoinverse and Laplacian matrix to structurally regularize the TM variables. Besides, in view of the fact that anomalies in traffic usually happen rarely and last briefly, sparsity-regularization is further implemented on traffic volume anomalies. This enables our models to jointly deal with the traffic monitoring issues of TM estimation and anomaly detection. The SRTM model is designed for static routing configurations, and the online traffic monitoring model DSRTM is presented for dynamic settings. In DSRTM, a forgetting parameter is introduced to incorporate information from both the latest and previous estimations, getting rid of the problem that storing a large amount of traffic data is a huge burden for computers. Furthermore, real-time monitoring enables DSRTM to be applied in scenarios that usually experience nonstationary. Efficient algorithms that are based on accelerated proximal gradient, gradient descent, and block coordinate descent methods are proposed to solve the related SRTM and DSRTM optimization problems, with experiments in synthetic and real networks under both static and dynamic routing configurations, verifying their feasibility and effectiveness.
---
paper_title: Integrated Anomaly Detection for cyber security of the substations

paper_content:

Cyber intrusions to substations of a power grid are a source of vulnerability since most substations are unmanned and with limited protection of the physical security. In the worst case, simultaneous intrusions into multiple substations can lead to severe cascading events, causing catastrophic power outages. In this paper, an integrated Anomaly Detection System (ADS) is proposed which contains host- and network-based anomaly detection systems for the substations, and simultaneous anomaly detection for multiple substations. Potential scenarios of simultaneous intrusions into the substations have been simulated using a substation automation testbed. The host-based anomaly detection considers temporal anomalies in the substation facilities, e.g., userinterfaces, Intelligent Electronic Devices (IEDs) and circuit breakers. The malicious behaviors of substation automation based on multicast messages, e.g., Generic Object Oriented Substation Event (GOOSE) and Sampled Measured Value (SMV), are incorporated in the proposed network-based anomaly detection. The proposed simultaneous intrusion detection method is able to identify the same type of attacks at multiple substations and their locations. The result is a new integrated tool for detection and mitigation of cyber intrusions at a single substation or multiple substations of a power grid.
---
paper_title: Distributed Anomaly Detection Using Minimum Volume Elliptical Principal Component Analysis

paper_content:

Principal component analysis and the residual error is an effective anomaly detection technique. In an environment where anomalies are present in the training set, the derived principal components can be skewed by the anomalies. A further aspect of anomaly detection is that data might be distributed across different nodes in a network and their communication to a centralized processing unit is prohibited due to communication cost. Current solutions to distributed anomaly detection rely on a hierarchical network infrastructure to aggregate data or models; however, in this environment, links close to the root of the tree become critical and congested. In this paper, an algorithm is proposed that is more robust in its derivation of the principal components of a training set containing anomalies. A distributed form of the algorithm is then derived where each node in a network can iterate towards the centralized solution by exchanging small matrices with neighboring nodes. Experimental evaluations on both synthetic and real-world data sets demonstrate the superior performance of the proposed approach in comparison to principal component analysis and alternative anomaly detection techniques. In addition, it is shown that in a variety of network infrastructures, the distributed form of the anomaly detection model is able to derive a close approximation of the centralized model.
---
paper_title: Multi-observer privacy-preserving Hidden Markov Models

paper_content:

Detection of malicious traffic and network health problems would be much easier if ISPs shared their data. Unfortunately, they are reluctant to share because doing so would either violate privacy legislation or expose business secrets. However, secure distributed computation allows calculations to be made using private data, without leaking this data. This paper presents such a method, allowing multiple parties to jointly infer a Hidden Markov Model (HMM) for traffic and/or user behaviour in order to detect anomalies. We extend prior work on HMMs in network security to include observations from multiple ISPs and develop secure protocols to infer the model parameters without revealing the private data. We implement a prototype of the protocols, and our experiments with the prototype show its has a reasonable computational and communications overhead, making it practical for adoption by ISPs.
---
paper_title: A Family of Joint Sparse PCA Algorithms for Anomaly Localization in Network Data Streams

paper_content:

Determining anomalies in data streams that are collected and transformed from various types of networks has recently attracted significant research interest. Principal component analysis (PCA) has been extensively applied to detecting anomalies in network data streams. However, none of existing PCA-based approaches addresses the problem of identifying the sources that contribute most to the observed anomaly, or anomaly localization. In this paper, we propose novel sparse PCA methods to perform anomaly detection and localization for network data streams. Our key observation is that we can localize anomalies by identifying a sparse low-dimensional space that captures the abnormal events in data streams. To better capture the sources of anomalies, we incorporate the structure information of the network stream data in our anomaly localization framework. Furthermore, we extend our joint sparse PCA framework with multidimensional Karhunen Loeve Expansion that considers both spatial and temporal domains of data streams to stabilize localization performance. We have performed comprehensive experimental studies of the proposed methods and have compared our methods with the state-of-the-art using three real-world data sets from different application domains. Our experimental studies demonstrate the utility of the proposed methods.
---
paper_title: A survey of network anomaly detection techniques

paper_content:

Information and Communication Technology (ICT) has a great impact on social wellbeing, economic growth and national security in todays world. Generally, ICT includes computers, mobile communication devices and networks. ICT is also embraced by a group of people with malicious intent, also known as network intruders, cyber criminals, etc. Confronting these detrimental cyber activities is one of the international priorities and important research area. Anomaly detection is an important data analysis task which is useful for identifying the network intrusions. This paper presents an in-depth analysis of four major categories of anomaly detection techniques which include classification, statistical, information theory and clustering. The paper also discusses research challenges with the datasets used for network intrusion detection. HighlightsMaps different types of anomalies with network attacks.Provides an up-to-date taxonomy of network anomaly detection.Evaluates effectiveness of different categories of techniques.Explores recent research related to publicly available network intrusion evaluation datasets.
---
paper_title: Network Anomaly Detection: Methods, Systems and Tools

paper_content:

Network anomaly detection is an important and dynamic research area. Many network intrusion detection methods and systems (NIDS) have been proposed in the literature. In this paper, we provide a structured and comprehensive overview of various facets of network anomaly detection so that a researcher can become quickly familiar with every aspect of network anomaly detection. We present attacks normally encountered by network intrusion detection systems. We categorize existing network anomaly detection methods and systems based on the underlying computational techniques used. Within this framework, we briefly describe and compare a large number of network anomaly detection methods and systems. In addition, we also discuss tools that can be used by network defenders and datasets that researchers in network anomaly detection can use. We also highlight research directions in network anomaly detection.
---
paper_title: Ensembles of incremental learners to detect anomalies in ad hoc sensor networks

paper_content:

Display Omitted In the past decade, rapid technological advances in the fields of electronics and telecommunications have given rise to versatile, ubiquitous decentralized embedded sensor systems with ad hoc wireless networking capabilities. Typically these systems are used to gather large amounts of data, while the detection of anomalies (such as system failures, intrusion, or unanticipated behavior of the environment) in the data (or other types or processing) is performed in centralized computer systems. In spite of the great interest that it attracts, the systematic porting and analysis of centralized anomaly detection algorithms to a decentralized paradigm (compatible with the aforementioned sensor systems) has not been thoroughly addressed in the literature. We approach this task from a new angle, assessing the viability of localized (in-node) anomaly detection based on machine learning. The main challenges we address are: (1) deploying decentralized, automated, online learning, anomaly detection algorithms within the stringent constraints of typical embedded systems; and (2) evaluating the performance of such algorithms and comparing them with that of centralized ones. To this end, we first analyze (and port) single and multi-dimensional input classifiers that are trained incrementally online and whose computational requirements are compatible with the limitations of embedded platforms. Next, we combine multiple classifiers in a single online ensemble. Then, using both synthetic and real-world datasets from different application domains, we extensively evaluate the anomaly detection performance of our algorithms and ensemble, in terms of precision and recall, and compare it to that of well-known offline, centralized machine learning algorithms. Our results show that the ensemble performs better than each individual decentralized classifier and that it can match the performance of the offline alternatives, thus showing that our approach is a viable solution to detect anomalies, even in environments with little a priori knowledge.
---
paper_title: Network Anomaly Traffic Detection Method Based on Support Vector Machine

paper_content:

Network traffic is the amount of data moving across a network at a specific time, and anomaly traffic detection is of great importance in computer networks. In this paper, we integrate the entropy theory and support vector machine to detect Network anomaly traffic. We utilize the entropy theory to establish the network traffic feature vector using the entropy theory, and then exploit the support vector machine to detect the network anomaly traffic by tackling a classification problem. Particularly, six types of network features are used to construct feature vector in this work, such as Source IP, Destination IP, Source Port, Destination Port, Packet Size and Packet Type. Afterwards, we provide the network feature vector to SVM to learn network traffic behaviors. Experimental results demonstrate that compared with existing method, our proposed method can detect network anomaly traffic with high accuracy.
---
paper_title: A Survey of Outlier Detection Methods in Network Anomaly Identification

paper_content:

The detection of outliers has gained considerable interest in data mining with the realization that outliers can be the key discovery to be made from very large databases. Outliers arise due to various reasons such as mechanical faults, changes in system behavior, fraudulent behavior, human error and instrument error. Indeed, for many applications the discovery of outliers leads to more interesting and useful results than the discovery of inliers. Detection of outliers can lead to identification of system faults so that administrators can take preventive measures before they escalate. It is possible that anomaly detection may enable detection of new attacks. Outlier detection is an important anomaly detection approach. In this paper, we present a comprehensive survey of well-known distance-based, density-based and other techniques for outlier detection and compare them. We provide definitions of outliers and discuss their detection based on supervised and unsupervised learning in the context of network anomaly detection.
---
paper_title: High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

paper_content:

High-dimensional problem domains pose significant challenges for anomaly detection. The presence of irrelevant features can conceal the presence of anomalies. This problem, known as the 'curse of dimensionality', is an obstacle for many anomaly detection techniques. Building a robust anomaly detection model for use in high-dimensional spaces requires the combination of an unsupervised feature extractor and an anomaly detector. While one-class support vector machines are effective at producing decision surfaces from well-behaved feature vectors, they can be inefficient at modelling the variation in large, high-dimensional datasets. Architectures such as deep belief networks (DBNs) are a promising technique for learning robust features. We present a hybrid model where an unsupervised DBN is trained to extract generic underlying features, and a one-class SVM is trained from the features learned by the DBN. Since a linear kernel can be substituted for nonlinear ones in our hybrid model without loss of accuracy, our model is scalable and computationally efficient. The experimental results show that our proposed model yields comparable anomaly detection performance with a deep autoencoder, while reducing its training and testing time by a factor of 3 and 1000, respectively. HighlightsWe use a combination of a one-class SVM and deep learning.In our model linear kernels can be used rather than nonlinear ones.Our model delivers a comparable accuracy with a deep autoencoder.Our model executes 3times faster in training and 1000 faster than a deep autoencoder.
---
paper_title: Outlier Detection Techniques for Wireless Sensor Networks: A Survey

paper_content:

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.
---
paper_title: Outliers detection and classification in wireless sensor networks

paper_content:

Abstract In the past few years, many wireless sensor networks had been deployed in the real world to collect large amounts of raw sensed data. However, the key challenge is to extract high-level knowledge from such raw data. In the applications of sensor networks, outlier/anomaly detection has been paid more and more attention. Outlier detection can be used to filter noisy data, find faulty nodes, and discover interesting events. In this paper we propose a novel in-network knowledge discovery approach that provides outlier detection and data clustering simultaneously. Our approach is capable to distinguish between an error due to faulty sensor and an error due to an event (probably an environmental event) which characterize the spatial and temporal correlations between events observed by sensor nodes in a confined network neighborhood. Experiments on both synthetic and real datasets show that the proposed algorithm outperforms other techniques in both effectiveness and efficiency.
---
paper_title: K-Means+ID3: A Novel Method for Supervised Anomaly Detection by Cascading K-Means Clustering and ID3 Decision Tree Learning Methods

paper_content:

In this paper, we present "k-means+ID3", a method to cascade k-means clustering and the ID3 decision tree learning methods for classifying anomalous and normal activities in a computer network, an active electronic circuit, and a mechanical mass-beam system. The k-means clustering method first partitions the training instances into k clusters using Euclidean distance similarity. On each cluster, representing a density region of normal or anomaly instances, we build an ID3 decision tree. The decision tree on each cluster refines the decision boundaries by learning the subgroups within the cluster. To obtain a final decision on classification, the decisions of the k-means and ID3 methods are combined using two rules: 1) the nearest-neighbor rule and 2) the nearest-consensus rule. We perform experiments on three data sets: 1) network anomaly data (NAD), 2) Duffing equation data (DED), and 3) mechanical system data (MSD), which contain measurements from three distinct application domains of computer networks, an electronic circuit implementing a forced Duffing equation, and a mechanical system, respectively. Results show that the detection accuracy of the k-means+ID3 method is as high as 96.24 percent at a false-positive-rate of 0.03 percent on NAD; the total accuracy is as high as 80.01 percent on MSD and 79.9 percent on DED
---
paper_title: A review on outlier detection techniques on data stream by using different approaches of K-Means algorithm

paper_content:

Data Stream mining has gained attraction from many researchers as there is need to mine large dataset which pose different challenges for researchers. Stream data is different compared to normal data as they are continuously produced from different applications which impose different challenges like massive, infinite, concept drift for processing. An object that does not obey the behavior of normal data object is called outliers. Outlier detection is used in different applications like fraud detection, intrusion detection, track environmental changes, medical diagnosis so there is need to detect outliers from data streams. Various approaches are used for outlier detection. Some of them use K-Means algorithm for outlier detection in data streams which help to create a similar group or cluster of data points. Data stream clustering techniques are highly helpful to cluster similar data items in data streams and also to detect the outliers from them, so they are called cluster based outlier detection. K-means algorithm is partition based algorithm which is used for clustering datasets into number of clusters. It is most common and popular algorithm for clustering due to its simplicity and efficiency. Purpose of this paper is to review of different approaches of outlier detection which is used for K-Means algorithm for clustering dataset with some other methods. Different application areas of outlier detection are discussed in this paper.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: Big Data Analytics for User-Activity Analysis and User-Anomaly Detection in Mobile Wireless Network

paper_content:

The next generation wireless networks are expected to operate in fully automated fashion to meet the burgeoning capacity demand and to serve users with superior quality of experience. Mobile wireless networks can leverage spatio-temporal information about user and network condition to embed the system with end-to-end visibility and intelligence. Big data analytics has emerged as a promising approach to unearth meaningful insights and to build artificially intelligent models with assistance of machine learning tools. Utilizing aforementioned tools and techniques, this paper contributes in two ways. First, we utilize mobile network data (Big Data)—call detail record—to analyze anomalous behavior of mobile wireless network. For anomaly detection purposes, we use unsupervised clustering techniques namely k-means clustering and hierarchical clustering. We compare the detected anomalies with ground truth information to verify their correctness. From the comparative analysis, we observe that when the network experiences abruptly high (unusual) traffic demand at any location and time, it identifies that as anomaly. This helps in identifying regions of interest in the network for special action such as resource allocation, fault avoidance solution, etc. Second, we train a neural-network-based prediction model with anomalous and anomaly-free data to highlight the effect of anomalies in data while training/building intelligent models. In this phase, we transform our anomalous data to anomaly-free and we observe that the error in prediction, while training the model with anomaly-free data has largely decreased as compared to the case when the model was trained with anomalous data.
---
paper_title: A survey of network anomaly detection techniques

paper_content:

Information and Communication Technology (ICT) has a great impact on social wellbeing, economic growth and national security in todays world. Generally, ICT includes computers, mobile communication devices and networks. ICT is also embraced by a group of people with malicious intent, also known as network intruders, cyber criminals, etc. Confronting these detrimental cyber activities is one of the international priorities and important research area. Anomaly detection is an important data analysis task which is useful for identifying the network intrusions. This paper presents an in-depth analysis of four major categories of anomaly detection techniques which include classification, statistical, information theory and clustering. The paper also discusses research challenges with the datasets used for network intrusion detection. HighlightsMaps different types of anomalies with network attacks.Provides an up-to-date taxonomy of network anomaly detection.Evaluates effectiveness of different categories of techniques.Explores recent research related to publicly available network intrusion evaluation datasets.
---
paper_title: An anomalous behavior detection model in cloud computing

paper_content:

This paper proposes an anomalous behavior detection model based on cloud computing. Virtual Machines (VMs) are one of the key components of cloud Infrastructure as a Service (IaaS). The security of such VMs is critical to IaaS security. Many studies have been done on cloud computing security issues, but research into VM security issues, especially regarding VM network traffic anomalous behavior detection, remains inadequate. More and more studies show that communication among internal nodes exhibits complex patterns. Communication among VMs in cloud computing is invisible. Researchers find such issues challenging, and few solutions have been proposed-leaving cloud computing vulnerable to network attacks. This paper proposes a model that uses Software-Defined Networks (SDN) to implement traffic redirection. Our model can capture inter-VM traffic, detect known and unknown anomalous network behaviors, adopt hybrid techniques to analyze VM network behaviors, and control network systems. The experimental results indicate that the effectiveness of our approach is greater than 90%, and prove the feasibility of the model.
---
paper_title: Anomaly Detection Based on LRD Behavior Analysis of Decomposed Control and Data Planes Network Traffic Using SOSS and FARIMA Models

paper_content:

The detection of anomalies in network traffic, such as low volume attacks and abnormalities, has become a pressing problem in today’s large volume of Internet traffic. To this end, various anomaly detection techniques have been developed, including techniques based on long-range dependence (LRD) behavior estimation of network traffic. However, the existing LRD-based techniques analyze the aggregated WHOLE (control plus data) traffic, which might not be sufficient to detect short-duration and low-volume attacks and abnormalities in the traffic. This is because such anomalies might pass unnoticed in large volume of the normal background traffic. To address this issue, we propose a method that examines the LRD behavior of control and data planes traffic separately, which improves the detection efficacy. For LRD behavior analysis, the proposed method integrates the correlation structures of second-order self-similar and fractional autoregressive integrated moving average models. The performance of the proposed method is empirically evaluated and validated over a relatively recent real Internet traffic captured at King Saud University’s network. The analysis and results demonstrate that the proposed method efficiently detects such low volume and short duration attacks and abnormalities in the traffic, which would not be detected by merely analyzing the aggregated WHOLE traffic without decomposing it into control and data planes traffic.
---
paper_title: Wearable Sensor-Based Behavioral Anomaly Detection in Smart Assisted Living Systems

paper_content:

Detecting behavioral anomalies in human daily life is important to developing smart assisted-living systems for elderly care. Based on data collected from wearable motion sensors and the associated locational context, this paper presents a coherent anomaly detection framework to effectively detect different behavioral anomalies in human daily life. Four types of anomalies, including spatial anomaly, timing anomaly, duration anomaly, and sequence anomaly, are detected using a probabilistic theoretical framework. This framework is based on complex activity recognition using dynamic Bayesian network modeling. The maximum-likelihood estimation algorithm and Laplace smoothing are used in learning the parameters in the anomaly detection model. We conducted experimental evaluation in a mock apartment environment, and the results verified the effectiveness of the proposed framework. We expect that this behavioral anomaly detection system can be integrated into future smart homes for elderly care.
---
paper_title: A Survey of Outlier Detection Methods in Network Anomaly Identification

paper_content:

The detection of outliers has gained considerable interest in data mining with the realization that outliers can be the key discovery to be made from very large databases. Outliers arise due to various reasons such as mechanical faults, changes in system behavior, fraudulent behavior, human error and instrument error. Indeed, for many applications the discovery of outliers leads to more interesting and useful results than the discovery of inliers. Detection of outliers can lead to identification of system faults so that administrators can take preventive measures before they escalate. It is possible that anomaly detection may enable detection of new attacks. Outlier detection is an important anomaly detection approach. In this paper, we present a comprehensive survey of well-known distance-based, density-based and other techniques for outlier detection and compare them. We provide definitions of outliers and discuss their detection based on supervised and unsupervised learning in the context of network anomaly detection.
---
paper_title: Outlier Detection Techniques for Wireless Sensor Networks: A Survey

paper_content:

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.
---
paper_title: Distributed Real-Time Anomaly Detection in Networked Industrial Sensing Systems

paper_content:

Reliable real-time sensing plays a vital role in ensuring the reliability and safety of industrial cyberphysical systems (CPSs) such as wireless sensor and actuator networks. For many reasons, such as harsh industrial environments, fault-prone sensors, or malicious attacks, sensor readings may be abnormal or faulty. This could lead to serious system performance degradation or even catastrophic failure. Current anomaly detection approaches are either centralized and complicated or restricted due to strict assumptions, which are not suitable for practical large-scale networked industrial sensing systems (NISSs), where sensing devices are connected via digital communications, such as wireless sensor networks or smart grid systems. In this paper, we introduce a fully distributed general anomaly detection (GAD) scheme, which uses graph theory and exploits spatiotemporal correlations of physical processes to carry out real-time anomaly detection for general large-scale NISSs. We formally prove the scalability of our GAD approach and evaluate the performance of GAD for two industrial applications: building structure monitoring and smart grids. Extensive trace-driven simulations validate our theoretical analysis and demonstrate that our approach can significantly outperform state-of-the-art approaches in terms of detection accuracy and efficiency.
---
paper_title: Structure Regularized Traffic Monitoring for Traffic Matrix Estimation and Anomaly Detection by Link-Load Measurements

paper_content:

We consider problems of traffic matrix (TM) estimation and anomaly detection utilizing link-load traffic measurements. Models including structure regularized traffic monitoring (SRTM) and dynamic SRTM (DSRTM) are presented to realize traffic monitoring under static and dynamic routing configurations, respectively. Considering that real traffic data are usually approximately low-rank but exhibit strong spatial and temporal dependencies, we define spatial and temporal regularization matrices based on Moore–Penrose pseudoinverse and Laplacian matrix to structurally regularize the TM variables. Besides, in view of the fact that anomalies in traffic usually happen rarely and last briefly, sparsity-regularization is further implemented on traffic volume anomalies. This enables our models to jointly deal with the traffic monitoring issues of TM estimation and anomaly detection. The SRTM model is designed for static routing configurations, and the online traffic monitoring model DSRTM is presented for dynamic settings. In DSRTM, a forgetting parameter is introduced to incorporate information from both the latest and previous estimations, getting rid of the problem that storing a large amount of traffic data is a huge burden for computers. Furthermore, real-time monitoring enables DSRTM to be applied in scenarios that usually experience nonstationary. Efficient algorithms that are based on accelerated proximal gradient, gradient descent, and block coordinate descent methods are proposed to solve the related SRTM and DSRTM optimization problems, with experiments in synthetic and real networks under both static and dynamic routing configurations, verifying their feasibility and effectiveness.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: Network Anomaly Detection: Methods, Systems and Tools

paper_content:

Network anomaly detection is an important and dynamic research area. Many network intrusion detection methods and systems (NIDS) have been proposed in the literature. In this paper, we provide a structured and comprehensive overview of various facets of network anomaly detection so that a researcher can become quickly familiar with every aspect of network anomaly detection. We present attacks normally encountered by network intrusion detection systems. We categorize existing network anomaly detection methods and systems based on the underlying computational techniques used. Within this framework, we briefly describe and compare a large number of network anomaly detection methods and systems. In addition, we also discuss tools that can be used by network defenders and datasets that researchers in network anomaly detection can use. We also highlight research directions in network anomaly detection.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: Cooperative sensor anomaly detection using global information

paper_content:

Sensor networks are deployed in many application areas nowadays ranging from environment monitoring, industrial monitoring, and agriculture monitoring to military battlefield sensing. The accuracy of sensor readings is without a doubt one of the most important measures to evaluate the quality of a sensor and its network. Therefore, this work is motivated to propose approaches that can detect and repair erroneous (i.e., dirty) data caused by inevitable system problems involving various hardware and software components of sensor networks. As information about a single event of interest in a sensor network is usually reflected in multiple measurement points, the inconsistency among multiple sensor measurements serves as an indicator for data quality problem. The focus of this paper is thus to study methods that can effectively detect and identify erroneous data among inconsistent observations based on the inherent structure of various sensor measurement series from a group of sensors. Particularly, we present three models to characterize the inherent data structures among sensor measurement traces and then apply these models individually to guide the error detection of a sensor network. First, we propose a multivariate Gaussian model which explores the correlated data changes of a group of sensors. Second, we present a Principal Component Analysis (PCA) model which captures the sparse geometric relationship among sensors in a network. The PCA model is motivated by the fact that not all sensor networks have clustered sensor deployment and clear data correlation structure. Further, if the sensor data show non-linear characteristic, a traditional PCA model can not capture the data attributes properly. Therefore, we propose a third model which utilizes kernel functions to map the original data into a high dimensional feature space and then apply PCA model on the mapped linearized data. All these three models serve the purpose of capturing the underlying phenomenon of a sensor network from its global view, and then guide the error detection to discover any anomaly observations. We conducted simulations for each of the proposed models, and evaluated the performance by deriving the Receiver Operating Characteristic (ROC) curves.
---
paper_title: An anomalous behavior detection model in cloud computing

paper_content:

This paper proposes an anomalous behavior detection model based on cloud computing. Virtual Machines (VMs) are one of the key components of cloud Infrastructure as a Service (IaaS). The security of such VMs is critical to IaaS security. Many studies have been done on cloud computing security issues, but research into VM security issues, especially regarding VM network traffic anomalous behavior detection, remains inadequate. More and more studies show that communication among internal nodes exhibits complex patterns. Communication among VMs in cloud computing is invisible. Researchers find such issues challenging, and few solutions have been proposed-leaving cloud computing vulnerable to network attacks. This paper proposes a model that uses Software-Defined Networks (SDN) to implement traffic redirection. Our model can capture inter-VM traffic, detect known and unknown anomalous network behaviors, adopt hybrid techniques to analyze VM network behaviors, and control network systems. The experimental results indicate that the effectiveness of our approach is greater than 90%, and prove the feasibility of the model.
---
paper_title: Network anomaly detection using IP flows with Principal Component Analysis and Ant Colony Optimization

paper_content:

It is remarkable how proactive network management is in such demand nowadays, since networks are growing in size and complexity and Information Technology services cannot be stopped. In this manner, it is necessary to use an approach which proactively identifies traffic behavior patterns which may harm the network's normal operations. Aiming an automated management to detect and prevent potential problems, we present and compare two novel anomaly detection mechanisms based on statistical procedure Principal Component Analysis and the Ant Colony Optimization metaheuristic. These methods generate a traffic profile, called Digital Signature of Network Segment using Flow analysis (DSNSF), which is adopted as normal network behavior. Then, this signature is compared with the real network traffic by using a modification of the Dynamic Time Warping metric in order to recognize anomalous events. Thus, a seven-dimensional analysis of IP flows is performed, allowing the characterization of bits, packets and flows traffic transmitted per second, and the extraction of descriptive flow attributes, like source IP address, destination IP address, source TCP/UDP port and destination TCP/UDP port. The systems were evaluated using a real network environment and showed promising results. Moreover, the correspondence between true-positive and false-positive rates demonstrates that the systems are able to enhance the detection of anomalous behavior by maintaining a satisfactory false-alarm rate. Display Omitted Anomaly detection issue is addressed based on network traffic profiling.Proposal and comparison of detection methods belonging to distinct algorithm classes.Detection mechanism constructed over an adaptation of a pattern matching technique.Use of real and simulated traffic to evaluate the proposed methods.Traffic patterns that may harm the network operations are proactively identified.
---
paper_title: Effective Statistical Detection of Smart Confidentiality Attacks in Multi-Domain Networks

paper_content:

The need to preserve information confidentiality among network providers has prevented the actual deployment of effective Traffic Engineering (TE) solutions for QoS-enabled end-to-end connectivity services in multi-domain multi-provider networks. The use of Path Computation Element (PCE) architecture can foster the effective implementation of TE through a centralized engine devoted to end-to-end path computations. However, despite authentication, authorization and encryption, confidentiality issues may arise due to abuses of information included in path computation replies to bogus requests issued by malicious PCEs. This paper first demonstrates the security leak allowing the exposure of intra-domain information in current inter-PCE path computation procedures. Then it proposes an anomaly-based approach, namely PCE Anomaly Detector (PAD) in order to detect malicious utilization of path computation services. The proposed PAD employs a novel double-step multi-dimensional formulation based on the Sequential Hypothesis Testing (SHT) statistical classification procedure, able to recognize a suspicious sequence of requests while aiming at inferring confidential information about other domains. PAD is extensively evaluated through simulations. Results show good performance in terms of detection capabilities while guaranteeing the trade-off between accuracy and responsiveness, minimizing false alarm occurrences. Robustness against smart attacks is also proved with respect to a comprehensive set of attack patterns and under different network load conditions. Finally, intra-domain information exposition is evaluated, showing the PAD ability to preserve confidentiality.
---
paper_title: Segment-Based Anomaly Detection with Approximated Sample Covariance Matrix in Wireless Sensor Networks

paper_content:

In wireless sensor networks (WSNs), it has been observed that most abnormal events persist over a considerable period of time instead of being transient. As existing anomaly detection techniques usually operate in a point-based manner that handles each observation individually, they are unable to reliably and efficiently report such long-term anomalies appeared in an individual sensor node. Therefore, in this paper, we focus on a new technique for handling data in a segment-based manner. Considering a collection of neighbouring data segments as random variables, we determine those behaving abnormally by exploiting their spatial predictabilities and, motivated by spatial analysis, specifically investigate how to implement a prediction variance detector in a WSN. As the communication cost incurred in aggregating a covariance matrix is finally optimised using the Spearman’s rank correlation coefficient and differential compression, the proposed scheme is able to efficiently detect a wide range of long-term anomalies. In theory, comparing to the regular centralised approach, it can reduce the communication cost by approximately 80 percent. Moreover, its effectiveness is demonstrated by the numerical experiments, with a real world data set collected by the Intel Berkeley ResearchLab (IBRL).
---
paper_title: Efficient Computer Network Anomaly Detection by Changepoint Detection Methods

paper_content:

We consider the problem of efficient on-line anomaly detection in computer network traffic. The problem is approached statistically, as that of sequential (quickest) changepoint detection. A multi-cyclic setting of quickest change detection is a natural fit for this problem. We propose a novel score-based multi-cyclic detection algorithm. The algorithm is based on the so-called Shiryaev-Roberts procedure. This procedure is as easy to employ in practice and as computationally inexpensive as the popular Cumulative Sum chart and the Exponentially Weighted Moving Average scheme. The likelihood ratio based Shiryaev-Roberts procedure has appealing optimality properties, particularly it is exactly optimal in a multi-cyclic setting geared to detect a change occurring at a far time horizon. It is therefore expected that an intrusion detection algorithm based on the Shiryaev-Roberts procedure will perform better than other detection schemes. This is confirmed experimentally for real traces. We also discuss the possibility of complementing our anomaly detection algorithm with a spectral-signature intrusion detection system with false alarm filtering and true attack confirmation capability, so as to obtain a synergistic system.
---
paper_title: Mobile agent-based cross-layer anomaly detection in smart home sensor networks using fuzzy logic

paper_content:

Despite the rapid advancements in consumer electronics, the data transmitted by sensing devices in a smart home environment are still vulnerable to anomalies due to node faults, transmission errors, or attacks. This affects the reliability of the received sensed data and may lead to the incorrect decision making at both local (i.e., smart home) and global (i.e., smart city) levels. This study introduces a novel mobile agent-based cross-layer anomaly detection scheme, which takes into account stochastic variability in cross-layer data obtained from received data packets, and defines fuzzy logic-based soft boundaries to characterize behavior of sensor nodes. This cross-layer design approach empowers the proposed scheme to detect both node and link anomalies, and also effectively transmits mobile agents by considering the communication link-state before transmission of the mobile agent. The proposed scheme is implemented on a real testbed and a modular application software is developed to manage the anomaly detection system in the smart home. The experimental results show that the proposed scheme detects cross-layer anomalies with high accuracy and considerably reduces the energy consumption caused by the mobile agent transmission in the poor communication link-state situations.
---
paper_title: Wireless Anomaly Detection Based on IEEE 802.11 Behavior Analysis

paper_content:

Wireless communication networks are pervading every aspect of our lives due to their fast, easy, and inexpensive deployment. They are becoming ubiquitous and have been widely used to transfer critical information, such as banking accounts, credit cards, e-mails, and social network credentials. The more pervasive the wireless technology is going to be, the more important its security issue will be. Whereas the current security protocols for wireless networks have addressed the privacy and confidentiality issues, there are unaddressed vulnerabilities threatening their availability and integrity (e.g., denial of service, session hijacking, and MAC address spoofing attacks). In this paper, we describe an anomaly based intrusion detection system for the IEEE 802.11 wireless networks based on behavioral analysis to detect deviations from normal behaviors that are triggered by wireless network attacks. Our anomaly behavior analysis of the 802.11 protocols is based on monitoring the n-consecutive transitions of the protocol state machine. We apply sequential machine learning techniques to model the n-transition patterns in the protocol and characterize the probabilities of these transitions being normal. We have implemented several experiments to evaluate our system performance. By cross validating the system over two different wireless channels, we have achieved a low false alarm rate (<0.1%). We have also evaluated our approach against an attack library of known wireless attacks and has achieved more than 99% detection rate.
---
paper_title: Anomaly Detection Based on LRD Behavior Analysis of Decomposed Control and Data Planes Network Traffic Using SOSS and FARIMA Models

paper_content:

The detection of anomalies in network traffic, such as low volume attacks and abnormalities, has become a pressing problem in today’s large volume of Internet traffic. To this end, various anomaly detection techniques have been developed, including techniques based on long-range dependence (LRD) behavior estimation of network traffic. However, the existing LRD-based techniques analyze the aggregated WHOLE (control plus data) traffic, which might not be sufficient to detect short-duration and low-volume attacks and abnormalities in the traffic. This is because such anomalies might pass unnoticed in large volume of the normal background traffic. To address this issue, we propose a method that examines the LRD behavior of control and data planes traffic separately, which improves the detection efficacy. For LRD behavior analysis, the proposed method integrates the correlation structures of second-order self-similar and fractional autoregressive integrated moving average models. The performance of the proposed method is empirically evaluated and validated over a relatively recent real Internet traffic captured at King Saud University’s network. The analysis and results demonstrate that the proposed method efficiently detects such low volume and short duration attacks and abnormalities in the traffic, which would not be detected by merely analyzing the aggregated WHOLE traffic without decomposing it into control and data planes traffic.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: Network Anomaly Detection: Methods, Systems and Tools

paper_content:

Network anomaly detection is an important and dynamic research area. Many network intrusion detection methods and systems (NIDS) have been proposed in the literature. In this paper, we provide a structured and comprehensive overview of various facets of network anomaly detection so that a researcher can become quickly familiar with every aspect of network anomaly detection. We present attacks normally encountered by network intrusion detection systems. We categorize existing network anomaly detection methods and systems based on the underlying computational techniques used. Within this framework, we briefly describe and compare a large number of network anomaly detection methods and systems. In addition, we also discuss tools that can be used by network defenders and datasets that researchers in network anomaly detection can use. We also highlight research directions in network anomaly detection.
---
paper_title: A Survey of Outlier Detection Methods in Network Anomaly Identification

paper_content:

The detection of outliers has gained considerable interest in data mining with the realization that outliers can be the key discovery to be made from very large databases. Outliers arise due to various reasons such as mechanical faults, changes in system behavior, fraudulent behavior, human error and instrument error. Indeed, for many applications the discovery of outliers leads to more interesting and useful results than the discovery of inliers. Detection of outliers can lead to identification of system faults so that administrators can take preventive measures before they escalate. It is possible that anomaly detection may enable detection of new attacks. Outlier detection is an important anomaly detection approach. In this paper, we present a comprehensive survey of well-known distance-based, density-based and other techniques for outlier detection and compare them. We provide definitions of outliers and discuss their detection based on supervised and unsupervised learning in the context of network anomaly detection.
---
paper_title: Outlier Detection Techniques for Wireless Sensor Networks: A Survey

paper_content:

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.
---
paper_title: Cooperative sensor anomaly detection using global information

paper_content:

Sensor networks are deployed in many application areas nowadays ranging from environment monitoring, industrial monitoring, and agriculture monitoring to military battlefield sensing. The accuracy of sensor readings is without a doubt one of the most important measures to evaluate the quality of a sensor and its network. Therefore, this work is motivated to propose approaches that can detect and repair erroneous (i.e., dirty) data caused by inevitable system problems involving various hardware and software components of sensor networks. As information about a single event of interest in a sensor network is usually reflected in multiple measurement points, the inconsistency among multiple sensor measurements serves as an indicator for data quality problem. The focus of this paper is thus to study methods that can effectively detect and identify erroneous data among inconsistent observations based on the inherent structure of various sensor measurement series from a group of sensors. Particularly, we present three models to characterize the inherent data structures among sensor measurement traces and then apply these models individually to guide the error detection of a sensor network. First, we propose a multivariate Gaussian model which explores the correlated data changes of a group of sensors. Second, we present a Principal Component Analysis (PCA) model which captures the sparse geometric relationship among sensors in a network. The PCA model is motivated by the fact that not all sensor networks have clustered sensor deployment and clear data correlation structure. Further, if the sensor data show non-linear characteristic, a traditional PCA model can not capture the data attributes properly. Therefore, we propose a third model which utilizes kernel functions to map the original data into a high dimensional feature space and then apply PCA model on the mapped linearized data. All these three models serve the purpose of capturing the underlying phenomenon of a sensor network from its global view, and then guide the error detection to discover any anomaly observations. We conducted simulations for each of the proposed models, and evaluated the performance by deriving the Receiver Operating Characteristic (ROC) curves.
---
paper_title: Detecting Insider Threats Using RADISH: A System for Real-Time Anomaly Detection in Heterogeneous Data Streams

paper_content:

We present a scalable system for high-throughput real-time analysis of heterogeneous data streams. Our architecture enables incremental development of models for predictive analytics and anomaly detection as data arrives into the system. In contrast with batch data-processing systems, such as Hadoop, that can have high latency, our architecture allows for ingest and analysis of data on the fly, thereby detecting and responding to anomalous behavior in near real time. This timeliness is important for applications such as insider threat, financial fraud, and network intrusions. We demonstrate an application of this system to the problem of detecting insider threats, namely, the misuse of an organization's resources by users of the system and present results of our experiments on a publicly available insider threat dataset.
---
paper_title: Network Volume Anomaly Detection and Identification in Large-scale Networks based on Online Time-structured Traffic Tensor Tracking

paper_content:

This paper addresses network anomography, that is, the problem of inferring network-level anomalies from indirect link measurements. This problem is cast as a low-rank subspace tracking problem for normal flows under incomplete observations and an outlier detection problem for abnormal flows. Since traffic data is large-scale time-structured data accompanied with noise and outliers under partial observations, an efficient modeling method is essential. To this end, this paper proposes an online subspace tracking of a Hankelized time-structured traffic tensor for normal flows based on the Candecomp/PARAFAC decomposition exploiting the recursive least squares algorithm. We estimate abnormal flows as outlier sparse flows via sparsity maximization in the underlying under-constrained linear-inverse problem. A major advantage is that our algorithm estimates normal flows by low-dimensional matrices with time-directional features as well as the spatial correlation of multiple links without using the past observed measurements and the past model parameters. Extensive numerical evaluations show that the proposed algorithm achieves faster convergence per iteration of model approximation and better volume anomaly detection performance compared to state-of-the-art algorithms.
---
paper_title: Mobile Payment Anomaly Detection Mechanism Based on Information Entropy

paper_content:

With the rapid growth in the number of mobile phone users, mobile payments have become an important part of mobile e-commerce applications. Secure payment systems directly affect the security of e-commerce systems. This study proposes an anomaly detection mechanism supported by an information entropy method combined with neural network to improve mobile payments security. As the entropy value is sensitive and have much difference between normal and abnormal traffic flow in the mobile payment system, the abnormal traffic data will be detected. The simulation result shows that it can realise the effective monitoring of abnormal flow in the mobile payment system.
---
paper_title: Network-Wide Anomaly Event Detection and Diagnosis With perfSONAR

paper_content:

High-performance computing (HPC) environments supporting data-intensive applications need multidomain network performance measurements from open frameworks such as perfSONAR. Detected network-wide correlated anomaly events that impact data throughput performance need to be quickly and accurately notified along with a root-cause analysis for remediation. In this paper, we present a novel network anomaly events detection and diagnosis scheme for network-wide visibility that improves accuracy of root-cause analysis. We address analysis limitations in cases where there is absence of complete network topology information, and when measurement probes are mis-calibrated leading to erroneous diagnosis. Our proposed scheme fuses perfSONAR time-series path measurements data from multiple domains using principal component analysis (PCA) to transform data for accurate correlated and uncorrelated anomaly events detection. We quantify the certainty of such detection using a measurement data sanity checking that involves: 1) measurement data reputation analysis to qualify the measurement samples and 2) filter framework to prune potentially misleading samples. Lastly, using actual perfSONAR one-way delay measurement traces, we show our proposed scheme’s effectiveness in diagnosing the root-cause of critical network performance anomaly events.
---
paper_title: A Family of Joint Sparse PCA Algorithms for Anomaly Localization in Network Data Streams

paper_content:

Determining anomalies in data streams that are collected and transformed from various types of networks has recently attracted significant research interest. Principal component analysis (PCA) has been extensively applied to detecting anomalies in network data streams. However, none of existing PCA-based approaches addresses the problem of identifying the sources that contribute most to the observed anomaly, or anomaly localization. In this paper, we propose novel sparse PCA methods to perform anomaly detection and localization for network data streams. Our key observation is that we can localize anomalies by identifying a sparse low-dimensional space that captures the abnormal events in data streams. To better capture the sources of anomalies, we incorporate the structure information of the network stream data in our anomaly localization framework. Furthermore, we extend our joint sparse PCA framework with multidimensional Karhunen Loeve Expansion that considers both spatial and temporal domains of data streams to stabilize localization performance. We have performed comprehensive experimental studies of the proposed methods and have compared our methods with the state-of-the-art using three real-world data sets from different application domains. Our experimental studies demonstrate the utility of the proposed methods.
---
paper_title: Mobile agent-based cross-layer anomaly detection in smart home sensor networks using fuzzy logic

paper_content:

Despite the rapid advancements in consumer electronics, the data transmitted by sensing devices in a smart home environment are still vulnerable to anomalies due to node faults, transmission errors, or attacks. This affects the reliability of the received sensed data and may lead to the incorrect decision making at both local (i.e., smart home) and global (i.e., smart city) levels. This study introduces a novel mobile agent-based cross-layer anomaly detection scheme, which takes into account stochastic variability in cross-layer data obtained from received data packets, and defines fuzzy logic-based soft boundaries to characterize behavior of sensor nodes. This cross-layer design approach empowers the proposed scheme to detect both node and link anomalies, and also effectively transmits mobile agents by considering the communication link-state before transmission of the mobile agent. The proposed scheme is implemented on a real testbed and a modular application software is developed to manage the anomaly detection system in the smart home. The experimental results show that the proposed scheme detects cross-layer anomalies with high accuracy and considerably reduces the energy consumption caused by the mobile agent transmission in the poor communication link-state situations.
---
paper_title: Online and Scalable Unsupervised Network Anomaly Detection Method

paper_content:

Nowadays, network intrusion detectors mainly rely on knowledge databases to detect suspicious traffic. These databases have to be continuously updated which requires important human resources and time. Unsupervised network anomaly detectors overcome this issue by using “intelligent” techniques to identify anomalies without any prior knowledge. However, these systems are often very complex as they need to explore the network traffic to identify flows patterns. Therefore, they are often unable to meet real-time requirements. In this paper, we present a new online and real-time unsupervised network anomaly detection algorithm (ORUNADA). Our solution relies on a discrete time-sliding window to update continuously the feature space and an incremental grid clustering to detect rapidly the anomalies. The evaluations showed that ORUNADA can process online large network traffic while ensuring a low detection delay and good detection performance. The experiments performed on the traffic of a core network of a Spanish intermediate Internet service provider demonstrated that ORUNADA detects in less than half a second an anomaly after its occurrence. Furthermore, the results highlight that our solution outperforms in terms of true positive rate and false positive rate existing techniques reported in the literature.
---
paper_title: Wireless Anomaly Detection Based on IEEE 802.11 Behavior Analysis

paper_content:

Wireless communication networks are pervading every aspect of our lives due to their fast, easy, and inexpensive deployment. They are becoming ubiquitous and have been widely used to transfer critical information, such as banking accounts, credit cards, e-mails, and social network credentials. The more pervasive the wireless technology is going to be, the more important its security issue will be. Whereas the current security protocols for wireless networks have addressed the privacy and confidentiality issues, there are unaddressed vulnerabilities threatening their availability and integrity (e.g., denial of service, session hijacking, and MAC address spoofing attacks). In this paper, we describe an anomaly based intrusion detection system for the IEEE 802.11 wireless networks based on behavioral analysis to detect deviations from normal behaviors that are triggered by wireless network attacks. Our anomaly behavior analysis of the 802.11 protocols is based on monitoring the n-consecutive transitions of the protocol state machine. We apply sequential machine learning techniques to model the n-transition patterns in the protocol and characterize the probabilities of these transitions being normal. We have implemented several experiments to evaluate our system performance. By cross validating the system over two different wireless channels, we have achieved a low false alarm rate (<0.1%). We have also evaluated our approach against an attack library of known wireless attacks and has achieved more than 99% detection rate.
---
paper_title: Effective Statistical Detection of Smart Confidentiality Attacks in Multi-Domain Networks

paper_content:

The need to preserve information confidentiality among network providers has prevented the actual deployment of effective Traffic Engineering (TE) solutions for QoS-enabled end-to-end connectivity services in multi-domain multi-provider networks. The use of Path Computation Element (PCE) architecture can foster the effective implementation of TE through a centralized engine devoted to end-to-end path computations. However, despite authentication, authorization and encryption, confidentiality issues may arise due to abuses of information included in path computation replies to bogus requests issued by malicious PCEs. This paper first demonstrates the security leak allowing the exposure of intra-domain information in current inter-PCE path computation procedures. Then it proposes an anomaly-based approach, namely PCE Anomaly Detector (PAD) in order to detect malicious utilization of path computation services. The proposed PAD employs a novel double-step multi-dimensional formulation based on the Sequential Hypothesis Testing (SHT) statistical classification procedure, able to recognize a suspicious sequence of requests while aiming at inferring confidential information about other domains. PAD is extensively evaluated through simulations. Results show good performance in terms of detection capabilities while guaranteeing the trade-off between accuracy and responsiveness, minimizing false alarm occurrences. Robustness against smart attacks is also proved with respect to a comprehensive set of attack patterns and under different network load conditions. Finally, intra-domain information exposition is evaluated, showing the PAD ability to preserve confidentiality.
---
paper_title: Granular Evaluation of Anomalies in Wireless Sensor Networks Using Dynamic Data Partitioning with an Entropy Criteria

paper_content:

This paper presents an anomaly detection model that is granular and distributed to accurately and efficiently identify sensed data anomalies within wireless sensor networks. A more decentralised mechanism is introduced with wider use of in-network processing on a hierarchical sensor node topology resulting in a robust framework for dynamic data domains. This efficiently addresses the big data issue that is encountered in large scale industrial sensor network applications. Data vectors on each node’s observation domain is first partitioned using an unsupervised approach that is adaptive regarding dynamic data streams using cumulative point-wise entropy and average relative density . Second order statistical analysis applied on average relative densities and mean entropy values is then used to differentiate anomalies through robust and adaptive thresholds that are responsive to a dynamic environment. Anomaly detection is then performed in a non-parametric and non-probabilistic manner over the different network tiers in the hierarchical topology in offering increased granularity for evaluation. Experiments were performed extensively using both real and artificial data distributions representative of different dynamic and multi-density observation domains. Results demonstrate higher accuracies in detection as more than 94 percent accompanied by a desirable reduction of more than 85 percent in communication costs when compared to existing centralized methods.
---
paper_title: Wearable Sensor-Based Behavioral Anomaly Detection in Smart Assisted Living Systems

paper_content:

Detecting behavioral anomalies in human daily life is important to developing smart assisted-living systems for elderly care. Based on data collected from wearable motion sensors and the associated locational context, this paper presents a coherent anomaly detection framework to effectively detect different behavioral anomalies in human daily life. Four types of anomalies, including spatial anomaly, timing anomaly, duration anomaly, and sequence anomaly, are detected using a probabilistic theoretical framework. This framework is based on complex activity recognition using dynamic Bayesian network modeling. The maximum-likelihood estimation algorithm and Laplace smoothing are used in learning the parameters in the anomaly detection model. We conducted experimental evaluation in a mock apartment environment, and the results verified the effectiveness of the proposed framework. We expect that this behavioral anomaly detection system can be integrated into future smart homes for elderly care.
---
paper_title: Structure Regularized Traffic Monitoring for Traffic Matrix Estimation and Anomaly Detection by Link-Load Measurements

paper_content:

We consider problems of traffic matrix (TM) estimation and anomaly detection utilizing link-load traffic measurements. Models including structure regularized traffic monitoring (SRTM) and dynamic SRTM (DSRTM) are presented to realize traffic monitoring under static and dynamic routing configurations, respectively. Considering that real traffic data are usually approximately low-rank but exhibit strong spatial and temporal dependencies, we define spatial and temporal regularization matrices based on Moore–Penrose pseudoinverse and Laplacian matrix to structurally regularize the TM variables. Besides, in view of the fact that anomalies in traffic usually happen rarely and last briefly, sparsity-regularization is further implemented on traffic volume anomalies. This enables our models to jointly deal with the traffic monitoring issues of TM estimation and anomaly detection. The SRTM model is designed for static routing configurations, and the online traffic monitoring model DSRTM is presented for dynamic settings. In DSRTM, a forgetting parameter is introduced to incorporate information from both the latest and previous estimations, getting rid of the problem that storing a large amount of traffic data is a huge burden for computers. Furthermore, real-time monitoring enables DSRTM to be applied in scenarios that usually experience nonstationary. Efficient algorithms that are based on accelerated proximal gradient, gradient descent, and block coordinate descent methods are proposed to solve the related SRTM and DSRTM optimization problems, with experiments in synthetic and real networks under both static and dynamic routing configurations, verifying their feasibility and effectiveness.
---
paper_title: Integrated Anomaly Detection for cyber security of the substations

paper_content:

Cyber intrusions to substations of a power grid are a source of vulnerability since most substations are unmanned and with limited protection of the physical security. In the worst case, simultaneous intrusions into multiple substations can lead to severe cascading events, causing catastrophic power outages. In this paper, an integrated Anomaly Detection System (ADS) is proposed which contains host- and network-based anomaly detection systems for the substations, and simultaneous anomaly detection for multiple substations. Potential scenarios of simultaneous intrusions into the substations have been simulated using a substation automation testbed. The host-based anomaly detection considers temporal anomalies in the substation facilities, e.g., userinterfaces, Intelligent Electronic Devices (IEDs) and circuit breakers. The malicious behaviors of substation automation based on multicast messages, e.g., Generic Object Oriented Substation Event (GOOSE) and Sampled Measured Value (SMV), are incorporated in the proposed network-based anomaly detection. The proposed simultaneous intrusion detection method is able to identify the same type of attacks at multiple substations and their locations. The result is a new integrated tool for detection and mitigation of cyber intrusions at a single substation or multiple substations of a power grid.
---
paper_title: Sliding Modes for Anomaly Observation in TCP Networks: From Theory to Practice

paper_content:

Anomaly detection has been an active open problem in the networks community for several years. In this brief, we aim at detecting such abnormal signals by control theory techniques. Several classes of sliding mode observers are proposed for a fluid flow model of the transmission control protocol (TCP)/internet protocol network. Comparative simulations via network simulator NS-2 show the enhancement brought by a higher order sliding mode observer. The efficiency of this observer opens the way toward observing traffics with real TCP flow characteristics. To achieve this end, trace replay techniques for TCP traffic traces are presented. Finally, experiments lead to successful anomaly estimation under real traffic conditions.
---
paper_title: Multi-observer privacy-preserving Hidden Markov Models

paper_content:

Detection of malicious traffic and network health problems would be much easier if ISPs shared their data. Unfortunately, they are reluctant to share because doing so would either violate privacy legislation or expose business secrets. However, secure distributed computation allows calculations to be made using private data, without leaking this data. This paper presents such a method, allowing multiple parties to jointly infer a Hidden Markov Model (HMM) for traffic and/or user behaviour in order to detect anomalies. We extend prior work on HMMs in network security to include observations from multiple ISPs and develop secure protocols to infer the model parameters without revealing the private data. We implement a prototype of the protocols, and our experiments with the prototype show its has a reasonable computational and communications overhead, making it practical for adoption by ISPs.
---
paper_title: A Family of Joint Sparse PCA Algorithms for Anomaly Localization in Network Data Streams

paper_content:

Determining anomalies in data streams that are collected and transformed from various types of networks has recently attracted significant research interest. Principal component analysis (PCA) has been extensively applied to detecting anomalies in network data streams. However, none of existing PCA-based approaches addresses the problem of identifying the sources that contribute most to the observed anomaly, or anomaly localization. In this paper, we propose novel sparse PCA methods to perform anomaly detection and localization for network data streams. Our key observation is that we can localize anomalies by identifying a sparse low-dimensional space that captures the abnormal events in data streams. To better capture the sources of anomalies, we incorporate the structure information of the network stream data in our anomaly localization framework. Furthermore, we extend our joint sparse PCA framework with multidimensional Karhunen Loeve Expansion that considers both spatial and temporal domains of data streams to stabilize localization performance. We have performed comprehensive experimental studies of the proposed methods and have compared our methods with the state-of-the-art using three real-world data sets from different application domains. Our experimental studies demonstrate the utility of the proposed methods.
---
paper_title: Scalable Anomaly Detection for Smart City Infrastructure Networks

paper_content:

Dynamically detecting anomalies can be difficult in very large-scale infrastructure networks. The authors' approach addresses spatiotemporal anomaly detection in a smarter city context with large numbers of sensors deployed. They propose a scalable, hybrid Internet infrastructure for dynamically detecting potential anomalies in real time using stream processing. The infrastructure enables analytically inspecting and comparing anomalies globally using large-scale array processing. Deployed on a real pipe network topology of 1,891 nodes, this approach can effectively detect and characterize anomalies while minimizing the amount of data shared across the network.
---
paper_title: Combining heterogeneous anomaly detectors for improved software security

paper_content:

Abstract Host-based Anomaly Detection Systems (ADSs) monitor for significant deviations from normal software behavior. Several techniques have been investigated for detecting anomalies in system call sequences. Among these, Sequence Time-Delay Embedding (STIDE), Hidden Markov Model (HMM), and One-Class Support Vector Machine (OCSVM) have shown a high level of anomaly detection accuracy. Although ADSs can detect novel attacks, they generate a large number of false alarms due to the difficulty in obtaining complete descriptions of normal software behavior. This paper presents a multiple-detector ADS that efficiently combines the decisions from heterogeneous detectors (e.g., STIDE, HMM, and OCSVM), using Boolean combination in the Receiver Operating Characteristics (ROC) space, to reduce the false alarms. Results on two modern and large system call datasets generated from Linux and Windows operating systems show that the proposed ADS consistently outperforms an ADS based on a single best detector and on an ensemble of homogeneous detectors. At an operating point of zero percent alarm rate, the proposed multiple-detector ADS increased the true positive rate by 500% on the Linux dataset and by 25% on the Window dataset. Furthermore, the combinations of decisions from multiple heterogeneous detectors make the ADS more reliable and resilient against evasion and adversarial attacks.
---
paper_title: Wireless Anomaly Detection Based on IEEE 802.11 Behavior Analysis

paper_content:

Wireless communication networks are pervading every aspect of our lives due to their fast, easy, and inexpensive deployment. They are becoming ubiquitous and have been widely used to transfer critical information, such as banking accounts, credit cards, e-mails, and social network credentials. The more pervasive the wireless technology is going to be, the more important its security issue will be. Whereas the current security protocols for wireless networks have addressed the privacy and confidentiality issues, there are unaddressed vulnerabilities threatening their availability and integrity (e.g., denial of service, session hijacking, and MAC address spoofing attacks). In this paper, we describe an anomaly based intrusion detection system for the IEEE 802.11 wireless networks based on behavioral analysis to detect deviations from normal behaviors that are triggered by wireless network attacks. Our anomaly behavior analysis of the 802.11 protocols is based on monitoring the n-consecutive transitions of the protocol state machine. We apply sequential machine learning techniques to model the n-transition patterns in the protocol and characterize the probabilities of these transitions being normal. We have implemented several experiments to evaluate our system performance. By cross validating the system over two different wireless channels, we have achieved a low false alarm rate (<0.1%). We have also evaluated our approach against an attack library of known wireless attacks and has achieved more than 99% detection rate.
---
paper_title: An Efficient Network Log Anomaly Detection System Using Random Projection Dimensionality Reduction

paper_content:

Network traffic is increasing all the time and network services are becoming more complex and vulnerable. To protect these networks, intrusion detection systems are used. Signature-based intrusion detection cannot find previously unknown attacks, which is why anomaly detection is needed. However, many new systems are slow and complicated. We propose a log anomaly detection framework which aims to facilitate quick anomaly detection and also provide visualizations of the network traffic structure. The system preprocesses network logs into a numerical data matrix, reduces the dimensionality of this matrix using random projection and uses Mahalanobis distance to find outliers and calculate an anomaly score for each data point. Log lines that are too different are flagged as anomalies. The system is tested with real-world network data, and actual intrusion attempts are found. In addition, visualizations are created to represent the structure of the network data. We also perform computational time evaluation to ensure the performance is feasible. The system is fast, finds intrusion attempts and does not need clean training data.
---
paper_title: High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

paper_content:

High-dimensional problem domains pose significant challenges for anomaly detection. The presence of irrelevant features can conceal the presence of anomalies. This problem, known as the 'curse of dimensionality', is an obstacle for many anomaly detection techniques. Building a robust anomaly detection model for use in high-dimensional spaces requires the combination of an unsupervised feature extractor and an anomaly detector. While one-class support vector machines are effective at producing decision surfaces from well-behaved feature vectors, they can be inefficient at modelling the variation in large, high-dimensional datasets. Architectures such as deep belief networks (DBNs) are a promising technique for learning robust features. We present a hybrid model where an unsupervised DBN is trained to extract generic underlying features, and a one-class SVM is trained from the features learned by the DBN. Since a linear kernel can be substituted for nonlinear ones in our hybrid model without loss of accuracy, our model is scalable and computationally efficient. The experimental results show that our proposed model yields comparable anomaly detection performance with a deep autoencoder, while reducing its training and testing time by a factor of 3 and 1000, respectively. HighlightsWe use a combination of a one-class SVM and deep learning.In our model linear kernels can be used rather than nonlinear ones.Our model delivers a comparable accuracy with a deep autoencoder.Our model executes 3times faster in training and 1000 faster than a deep autoencoder.
---
paper_title: Outlier Detection Techniques for Wireless Sensor Networks: A Survey

paper_content:

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.
---
paper_title: A flow-based detection method for stealthy dictionary attacks against Secure Shell

paper_content:

Abstract SANS has warned about the new variants of SSH dictionary attacks that are very stealthy in comparison with a simple attack. In this paper, we propose a new method to detect simple and stealthy attacks by combining two key innovations. First, on the basis of our assumptions, we employ two criteria: “the existence of a connection protocol” and “the inter-arrival time of an auth-packet and the next” . These criteria are not available, though, owing to the confidentiality and flexibility of the SSH protocol. Second, we resolve this problem by identifying “the transition point of each sub-protocol” through flow features and machine learning algorithms. We evaluate the effectiveness through experiments on real network traffic at the edges in campus networks. The experimental results show that our method provides high accuracy with acceptable computational complexity.
---
paper_title: Hyperspherical cluster based distributed anomaly detection in wireless sensor networks

paper_content:

This article describes a distributed hyperspherical cluster based algorithm for identifying anomalies in measurements from a wireless sensor network, and an implementation on a real wireless sensor network testbed. The communication overhead incurred in the network is minimised by clustering sensor measurements and merging clusters before sending a compact description of the clusters to other nodes. An evaluation on several real and synthetic datasets demonstrates that the distributed hyperspherical cluster-based scheme achieves comparable detection accuracy with a significant reduction in communication overhead compared to a centralised scheme, where all the sensor node measurements are communicated to a central node for processing. Detecting anomalies in data is challenging on resource constrained networks.We propose a distributed algorithm using hyperspherical cluster based data models.The scheme is capable of identifying global anomalies at an individual node level.Comparable detection accuracy with significant reduction in communication overhead.Implemented and demonstrated on a real wireless sensor network test-bed.
---
paper_title: ENTROPY AND FLOW-BASED APPROACH FOR ANOMALOUS TRAFFIC FILTERING

paper_content:

Research Tools in Anomaly-based Intrusion Detection are highly dependent on appropriate traffic trace data. Traditional datasets present several issues such as: removal of sensitive information (anonymization) and insufficient number or volume of attack instances, which limit their quality for the design and evaluation of A-NIDSs. In this paper, we present a method for anomalous traffic filtering which can be used for generating anomaly-free traffic traces. The sanitized dataset can be used to improve the computation of the behaviour profiles during the training stage. The proposal is based on the construction and statistical analysis of the flow-level entropy space for the identification of outliers using three entropy estimators. Empirical results showed that the new traffic traces of the sanitized dataset have a distributional similarity among them greater than that presented among the original datasets.
---
paper_title: Detecting anomalies from big network traffic data using an adaptive detection approach

paper_content:

The unprecedented explosion of real-life big data sets have sparked a lot of research interests in data mining in recent years. Many of these big data sets are generated in network environment and are characterized by a dauntingly large size and high dimensionality which pose great challenges for detecting useful knowledge and patterns, such as network traffic anomalies, from them. In this paper, we study the problem of anomaly detection in big network connection data sets and propose an outlier detection technique, called Adaptive Stream Projected Outlier deTector (A-SPOT), to detect anomalies from large data sets using a novel adaptive subspace analysis approach. A case study of A-SPOT is conducted in this paper by deploying it to the 1999 KDD CUP anomaly detection application. Innovative approaches for training data generation, anomaly classification and false positive reduction are proposed in this paper as well to better tailor A-SPOT to deal with the case study. Experimental results demonstrate that A-SPOT is effective and efficient in detecting anomalies from network data sets and outperforms existing detection methods.
---
paper_title: FCAAIS: Anomaly based network intrusion detection through feature correlation analysis and association impact scale ☆

paper_content:

Abstract Due to the sensitivity of the information required to detect network intrusions efficiently, collecting huge amounts of network transactions is inevitable and the volume and details of network transactions available in recent years have been high. The meta-heuristic anomaly based assessment is vital in an exploratory analysis of intrusion related network transaction data. In order to forecast and deliver predictions about intrusion possibility from the available details of the attributes involved in network transaction. In this regard, a meta-heuristic assessment model called the feature correlation analysis and association impact scale is explored to estimate the degree of intrusion scope threshold from the optimal features of network transaction data available for training. With the motivation gained from the model called “network intrusion detection by feature association impact scale” that was explored in our earlier work, a novel and improved meta-heuristic assessment strategy for intrusion prediction is derived. In this strategy, linear canonical correlation for feature optimization is used and feature association impact scale is explored from the selected optimal features. The experimental result indicates that the feature correlation has a significant impact towards minimizing the computational and time complexity of measuring the feature association impact scale.
---
paper_title: Autonomous profile-based anomaly detection system using principal component analysis and flow analysis

paper_content:

An original anomaly detection system using principal component analysis is proposed.Our system was evaluated using real traffic data from a university.PCA proved effective in creating a digital signature of network traffic.Results pertaining to false alarm and accuracy rate are encouraging.Network anomalies were efficiently identified by our approach. Different techniques and methods have been widely used in the subject of automatic anomaly detection in computer networks. Attacks, problems and internal failures when not detected early may badly harm an entire Network system. Thus, an autonomous anomaly detection system based on the statistical method principal component analysis (PCA) is proposed. This approach creates a network profile called Digital Signature of Network Segment using Flow Analysis (DSNSF) that denotes the predicted normal behavior of a network traffic activity through historical data analysis. That digital signature is used as a threshold for volume anomaly detection to detect disparities in the normal traffic trend. The proposed system uses seven traffic flow attributes: bits, packets and number of flows to detect problems, and source and destination IP addresses and Ports, to provides the network administrator necessary information to solve them. Via evaluation techniques performed in this paper using real network traffic data, results showed good traffic prediction by the DSNSF and encouraging false alarm generation and detection accuracy on the detection schema using thresholds.
---

Title: Network anomaly detection research: a survey

Section 1: INTRODUCTION
Description 1: Introduce the importance of anomaly detection in information security, the major approaches to IDS, the focus of current research, and structure of the paper.

Section 2: METHODOLOGY
Description 2: Describe the systematic literature review (SLR) approach used in the study, including search criteria and the selection process of relevant articles.

Section 3: DISCUSSION
Description 3: Discuss the various aspects of anomaly detection studies, including methods or techniques, types of anomalies and attacks, output, data repositories, data types, and evaluation methods.

Section 4: Anomaly detection studies
Description 4: Summarize the existing research studies on network anomaly detection, highlighting the different techniques and models used to address various network anomalies.

Section 5: Network Categories and Application Domain
Description 5: Categorize the application network environments and domains where anomaly detection has been applied, along with the types of traffic or data used.

Section 6: Anomaly detection methods
Description 6: Detail the various methods used in anomaly detection research, such as clustering, classification, statistical methods, information theory, and more.

Section 7: Types of Anomaly
Description 7: Define the different types of anomalies detected in network traffic, including point, contextual, and collective anomalies, and map these to different network areas and attack types.

Section 8: Output of Anomalies
Description 8: Explain the techniques used to measure the capabilities of anomaly detection, including the use of scores and labels, and distinctions in measurement approaches for different network types.

Section 9: Data Repositories
Description 9: Describe the types of data repositories used in anomaly detection research, including real network captures, publicly available datasets, and testbed topologies.

Section 10: Data Type
Description 10: Discuss the types of input data used in anomaly detection, their attributes, and the relevance of univariate versus multivariate data analysis.

Section 11: Outlier/ Anomaly Identity
Description 11: Address the distinction between errors and events in outlier detection, and emphasize the importance of identifying sources of anomalies in different network contexts.

Section 12: Evaluation Method
Description 12: Outline the different evaluation and validation methods used by researchers to test their proposed anomaly detection systems, including experiments, simulations, and other approaches.

Section 13: Open Issues and Research Challenges
Description 13: Identify the main issues and challenges in network anomaly detection, such as detection capabilities, high dimensionality of data, computational complexity, and the need for real-time detection.

Section 14: CONCLUSION
Description 14: Summarize the findings of the survey, the current trends, and the future directions for research in network anomaly detection.