Review: False alarm minimization techniques in signature-based intrusion detection systems: A survey
---
paper_title: An Intrusion-Detection Model

paper_content:

A model of a real-time intrusion-detection expert system capable of detecting break-ins, penetrations, and other forms of computer abuse is described. The model is based on the hypothesis that security violations can be detected by monitoring a system's audit records for abnormal patterns of system usage. The model includes profiles for representing the behavior of subjects with respect to objects in terms of metrics and statistical models, and rules for acquiring knowledge about this behavior from audit records and for detecting anomalous behavior. The model is independent of any particular system, application environment, system vulnerability, or type of intrusion, thereby providing a framework for a general-purpose intrusion-detection expert system.
---
paper_title: Bro: A System for Detecting Network Intruders in Real-Time

paper_content:

Abstract We describe Bro, a stand-alone system for detecting network intruders in real-time by passively monitoring a network link over which the intruder's traffic transits. We give an overview of the system's design, which emphasizes high-speed (FDDI-rate) monitoring, real-time notification, clear separation between mechanism and policy, and extensibility. To achieve these ends, Bro is divided into an `event engine' that reduces a kernel-filtered network traffic stream into a series of higher-level events, and a `policy script interpreter' that interprets event handlers written in a specialized language used to express a site's security policy. Event handlers can update state information, synthesize new events, record information to disk, and generate real-time notifications via syslog. We also discuss a number of attacks that attempt to subvert passive monitoring systems and defenses against these, and give particulars of how Bro analyzes the six applications integrated into it so far: Finger, FTP, Portmapper, Ident, Telnet and Rlogin. The system is publicly available in source code form.
---
paper_title: Guide to Intrusion Detection and Prevention Systems (IDPS

paper_content:

This is a Hard copy of the NIST Special Publication 800-94 Guide to Intrusion Detection and Prevention Systems (IDPS). This publication describes the characteristics of IDPS technologies and provides recommendations for designing, implementing, configuring, securing, monitoring, and maintaining them. The types of IDPS technologies are differentiated primarily by the types of events that they monitor and the ways in which they are deployed. This publication discusses the following four types of IDPS technologies: Network-Based, which monitors network traffic for particular network segments or devices and analyzes the network and application protocol activity to identify suspicious activity Wireless, which monitors wireless network traffic and analyzes it to identify suspicious activity involving the wireless networking protocols themselves Network Behavior Analysis (NBA), which examines network traffic to identify threats that generate unusual traffic flows, such as distributed denial of service (DDoS) attacks, certain forms of malware, and policy violations (e.g., a client system providing network services to other systems) Host-Based, which monitors the characteristics of a single host and the events occurring within that host for suspicious activity. Implementing the following recommendations should facilitate more efficient and effective intrusion detection and prevention system use for Federal departments and agencies. Disclaimer This hardcopy is not published by National Institute of Standards and Technology (NIST), the US Government or US Department of Commerce. The publication of this document should not in any way imply any relationship or affiliation to the above named organizations and Government.
---
paper_title: Finding the Needle: Suppression of False Alarms in Large Intrusion Detection Data Sets

paper_content:

Managed security service providers (MSSPs) must manage and monitor thousands of intrusion detection sensors. The sensors often vary by manufacturer and software version, making the problem of creating generalized tools to separate true attacks from false positives particularly difficult. Often times it is useful from an operations perspective to know if a particular sensor is acting out of character. We propose a solution to this problem using anomaly detection techniques over the set of alarms produced by the sensors. Similar to the manner in which an anomaly based sensor detects deviations from normal user or system behavior, we establish the baseline behavior of a sensor and detect deviations from this baseline. We show that departures from this profile by a sensor have a high probability of being artifacts of genuine attacks. We evaluate a set of time-based Markovian heuristics against a simple compression algorithm and show that we are able to detect the existence of all attacks which were manually identified by security personnel, drastically reduce the number of false positives, and identify attacks which were overlooked during manual evaluation.
---
paper_title: Anomaly detection: A survey

paper_content:

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
---
paper_title: The Problem of False Alarms: Evaluation with Snort and DARPA 1999 Dataset

paper_content:

It is a common issue that an Intrusion Detection System (IDS) might generate thousand of alerts per day. The problem has got worse by the fact that IT infrastructure have become larger and more complicated, the number of generated alarms that need to be reviewed can escalate rapidly, making the task very difficult to manage. Moreover, a significant problem facing current IDS technology now is the high level of false alarms. The main purpose of this paper is to investigate the extent of false alarms problem in Snort, using the 1999 DARPA IDS evaluation dataset. A thorough investigation has been carried out to assess the accuracy of alerts generated by Snort IDS. Significantly, this experiment has revealed an unexpected result; with 69% of total generated alerts are considered to be false alarms.
---
paper_title: Using Adaptive Alert Classification to Reduce False Positives in Intrusion Detection

paper_content:

Intrusion Detection Systems (IDSs) are used to monitor computer systems for signs of security violations. Having detected such signs, IDSs trigger alerts to report them. These alerts are presented to a human analyst, who evaluates them and initiates an adequate response.
---
paper_title: The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection

paper_content:

Research in automated computer security intrusion detection, intrusion detection for short, is maturing. Several difficulties remain to be solved before intrusion detection systems become commonplace as part of real-world security solutions. One such difficulty regards the subject of effectiveness, how successful the intrusion detection system is at actually detecting intrusions with a high degree of certainty. With this as its starting point, this paper discusses the “base-rate fallacy” and how it influences the relative success of an intrusion detection system, under a set of reasonable circumstances. The conclusion is reached that the false-alarm rate quickly becomes a limiting factor.
---
paper_title: Alert correlation survey: framework and techniques

paper_content:

Managing raw alerts generated by various sensors are becoming of more significance to intrusion detection systems as more sensors with different capabilities are distributed spatially in the network. Alert Correlation addresses this issue by reducing, fusing and correlating raw alerts to provide a condensed, yet more meaningful view of the network from the intrusion standpoint. Techniques from a divers range of disciplines have been used by researchers for different aspects of correlation. This paper provides a survey of the state of the art in alert correlation techniques. Our main contribution is a two-fold classification of literature based on correlation framework and applied techniques. The previous works in each category have been described alongside with their strengths and weaknesses from our viewpoint.
---
paper_title: A survey of coordinated attacks and collaborative intrusion detection

paper_content:

Coordinated attacks, such as large-scale stealthy scans, worm outbreaks and distributed denial-of-service (DDoS) attacks, occur in multiple networks simultaneously. Such attacks are extremely difficult to detect using isolated intrusion detection systems (IDSs) that monitor only a limited portion of the Internet. In this paper, we summarize the current research directions in detecting such attacks using collaborative intrusion detection systems (CIDSs). In particular, we highlight two main challenges in CIDS research: CIDS architectures and alert correlation algorithms. We review the current CIDS approaches in terms of these two challenges. We conclude by highlighting opportunities for an integrated solution to large-scale collaborative intrusion detection.
---
paper_title: Enhancing byte-level network intrusion detection signatures with context

paper_content:

Many network intrusion detection systems (NIDS) use byte sequences as signatures to detect malicious activity. While being highly efficient, they tend to suffer from a high false-positive rate. We develop the concept of contextual signatures as an improvement of string-based signature-matching. Rather than matching fixed strings in isolation, we augment the matching process with additional context. When designing an efficient signature engine for the NIDS bro, we provide low-level context by using regular expressions for matching, and high-level context by taking advantage of the semantic information made available by bro's protocol analysis and scripting language. Therewith, we greatly enhance the signature's expressiveness and hence the ability to reduce false positives. We present several examples such as matching requests with replies, using knowledge of the environment, defining dependencies between signatures to model step-wise attacks, and recognizing exploit scans.To leverage existing efforts, we convert the comprehensive signature set of the popular freeware NIDS snort into bro's language. While this does not provide us with improved signatures by itself, we reap an established base to build upon. Consequently, we evaluate our work by comparing to snort, discussing in the process several general problems of comparing different NIDSs.
---
paper_title: Model-driven, network-context sensitive intrusion detection

paper_content:

Intrusion Detection Systems (IDSs) have the reputation of generating many false positives. Recent approaches, known as stateful IDSs, take the state of communication sessions into account to address this issue. A substantial reduction of false positives, however, requires some correlation between the state of the session, known vulnerabilities, and the gathering of more network context information by the IDS than what is currently done (e.g., configuration of a node, its operating system, running applications). In this paper we present an IDS approach that attempts to decrease the number of false positives by collecting more network context and combining this information with known vulnerabilities. The approach is model-driven as it relies on the modeling of packet and network information as UML class diagrams, and the definition of intrusion detection rules as OCL expressions constraining these diagrams. The approach is evaluated using real attacks on real systems, and appears to be promising.
---
paper_title: STATL: An Attack Language for State-based Intrusion Detection

paper_content:

STATL is an extensible state/transition-based attack description language designed to support intrusion detection. The language allows one to describe computer penetrations as sequences of actions that an attacker performs to compromise a computer system. A STATL description of an attack scenario can be used by an intrusion detection system to analyze a stream of events and detect possible ongoing intrusions. Since intrusion detection is performed in different domains (i.e., the network or the hosts) and in different operating environments (e.g., Linux, Solaris, or Windows NT), it is useful to have an extensible language that can be easily tailored to different target environments. STATL defines domain-independent features of attack scenarios and provides constructs for extending the language to describe attacks in particular domains and environments. The STATL language has been successfully used in describing both network-based and host-based attacks, and it has been tailored to very different environments, e.g., Sun Microsystems' Solaris and Microsoft's Windows NT. An implementation of the runtime support for the STATL language has been developed and a toolset of intrusion detection systems based on STATL has been implemented. The toolset was used in a recent intrusion detection evaluation effort, delivering very favorable results. This paper presents the details of the STATL syntax and its semantics. Real examples from both the host and network-based extensions of the language are also presented.
---
paper_title: A stateful intrusion detection system for World-Wide Web servers

paper_content:

Web servers are ubiquitous, remotely accessible, and often misconfigured. In addition, custom Web-based applications may introduce vulnerabilities that are overlooked even by the most security-conscious server administrators. Consequently, Web servers are a popular target for hackers. To mitigate the security exposure associated with Web servers, intrusion detection systems are deployed to analyze and screen incoming requests. The goal is to perform early detection of malicious activity and possibly prevent more serious damage to the protected site. Even though intrusion detection is critical for the security of Web servers, the intrusion detection systems available today only perform very simple analyses and are often vulnerable to simple evasion techniques. In addition, most systems do not provide sophisticated attack languages that allow a system administrator to specify custom, complex attack scenarios to be detected. We present WebSTAT, an intrusion detection system that analyzes Web requests looking for evidence of malicious behavior. The system is novel in several ways. First of all, it provides a sophisticated language to describe multistep attacks in terms of states and transitions. In addition, the modular nature of the system supports the integrated analysis of network traffic sent to the server host, operating system-level audit data produced by the server host, and the access logs produced by the Web server. By correlating different streams of events, it is possible to achieve more effective detection of Web-based attacks.
---
paper_title: Stateful intrusion detection for high-speed network's

paper_content:

As networks become faster there is an emerging need for security, analysis techniques that can keep up with the increased network throughput. Existing network-based intrusion detection sensors can barely, keep up with bandwidths of a few hundred Mbps. Analysis tools that can deal with higher throughput are unable to maintain state between different steps of an attack or they are limited to the analysis of packet headers. We propose a partitioning approach to network security, analysis that supports in-depth, stateful intrusion detection on high-speed links. The approach is centered around a slicing mechanism that divides the overall network traffic into subsets of manageable size. The traffic partitioning is done so that a single slice contains all the evidence necessary to detect a specific attack, making sensor-to-sensor interactions unnecessary. This paper describes the approach and presents a first experimental evaluation of its effectiveness.
---
paper_title: The NIDS cluster: Scalable, stateful network intrusion detection on commodity hardware

paper_content:

In this work we present a NIDS cluster as a scalable solution for realizing high-performance, stateful network intrusion detection on commodity hardware. The design addresses three challenges: (i) distributing traffic evenly across an extensible set of analysis nodes in a fashion that minimizes the communication required for coordination, (ii) adapting the NIDS's operation to support coordinating its low-level analysis rather than just aggregating alerts; and (iii) validating that the cluster produces sound results. Prototypes of our NIDS cluster now operate at the Lawrence Berkeley National Laboratory and the University of California at Berkeley. In both environments the clusters greatly enhance the power of the network security monitoring.
---
paper_title: NetShield: massive semantics-based vulnerability signature matching for high-speed networks

paper_content:

Accuracy and speed are the two most important metrics for Network Intrusion Detection/Prevention Systems (NIDS/NIPSes). Due to emerging polymorphic attacks and the fact that in many cases regular expressions (regexes) cannot capture the vulnerability conditions accurately, the accuracy of existing regex-based NIDS/NIPS systems has become a serious problem. In contrast, the recently-proposed vulnerability signatures (a.k.a data patches) can exactly describe the vulnerability conditions and achieve better accuracy. However, how to efficiently apply vulnerability signatures to high speed NIDS/NIPS with a large ruleset remains an untouched but challenging issue. This paper presents the first systematic design of vulnerability signature based parsing and matching engine, NetShield, which achieves multi-gigabit throughput while offering much better accuracy. Particularly, we made the following contributions: (i) we proposed a candidate selection algorithm which efficiently matches thousands of vulnerability signatures simultaneously requiring a small amount of memory; (ii) we proposed an automatic lightweight parsing state machine achieving fast protocol parsing. Experimental results show that the core engine of NetShield achieves at least 1.9+Gbps signature matching throughput on a 3.8GHz single-core PC, and can scale-up to at least 11+Gbps under a 8-core machine for 794 HTTP vulnerability signatures.
---
paper_title: Shield: vulnerability-driven network filters for preventing known vulnerability exploits

paper_content:

Software patching has not been effective as a first-line defense against large-scale worm attacks, even when patches have long been available for their corresponding vulnerabilities. Generally, people have been reluctant to patch their systems immediately, because patches are perceived to be unreliable and disruptive to apply. To address this problem, we propose a first-line worm defense in the network stack, using shields -- vulnerability-specific, exploit-generic network filters installed in end systems once a vulnerability is discovered, but before a patch is applied. These filters examine the incoming or outgoing traffic of vulnerable applications, and correct traffic that exploits vulnerabilities. Shields are less disruptive to install and uninstall, easier to test for bad side effects, and hence more reliable than traditional software patches. Further, shields are resilient to polymorphic or metamorphic variations of exploits [43].In this paper, we show that this concept is feasible by describing a prototype Shield framework implementation that filters traffic above the transport layer. We have designed a safe and restrictive language to describe vulnerabilities as partial state machines of the vulnerable application. The expressiveness of the language has been verified by encoding the signatures of several known vulnerabilites. Our evaluation provides evidence of Shield's low false positive rate and small impact on application throughput. An examination of a sample set of known vulnerabilities suggests that Shield could be used to prevent exploitation of a substantial fraction of the most dangerous ones.
---
paper_title: Alarm clustering for intrusion detection systems in computer networks

paper_content:

Until recently, network administrators manually arranged alarms produced by Intrusion Detection Systems (IDSs) to attain a high-level description of threats. As the number of alarms is increasingly growing, automatic tools for alarm clustering have been proposed to provide such a high level description of the attack scenario. In addition, it has been shown that effective threat analysis require the fusion of different sources of information, such as different IDSs, firewall logs, etc. In this paper, we propose a new strategy to perform alarm clustering which produces unified descriptions of attacks from multiple alarms. Tests have been performed on a live network where commercial and open-source IDSs analyzed network traffic.
---
paper_title: Mining intrusion detection alarms for actionable knowledge

paper_content:

In response to attacks against enterprise networks, administrators increasingly deploy intrusion detection systems. These systems monitor hosts, networks, and other resources for signs of security violations. The use of intrusion detection has given rise to another difficult problem, namely the handling of a generally large number of alarms. In this paper, we mine historical alarms to learn how future alarms can be handled more efficiently. First, we investigate episode rules with respect to their suitability in this approach. We report the difficulties encountered and the unexpected insights gained. In addition, we introduce a new conceptual clustering technique, and use it in extensive experiments with real-world data to show that intrusion detection alarms can be handled efficiently by using previously mined knowledge.
---
paper_title: Clustering intrusion detection alarms to support root cause analysis

paper_content:

It is a well-known problem that intrusion detection systems overload their human operators by triggering thousands of alarms per day. This paper presents a new approach for handling intrusion detection alarms more efficiently. Central to this approach is the notion that each alarm occurs for a reason, which is referred to as the alarm's root causes. This paper observes that a few dozens of rather persistent root causes generally account for over 90p of the alarms that an intrusion detection system triggers. Therefore, we argue that alarms should be handled by identifying and removing the most predominant and persistent root causes. To make this paradigm practicable, we propose a novel alarm-clustering method that supports the human analyst in identifying root causes. We present experiments with real-world intrusion detection alarms to show how alarm clustering helped us identify root causes. Moreover, we show that the alarm load decreases quite substantially if the identified root causes are eliminated so that they can no longer trigger alarms in the future.
---
paper_title: IDS false alarm filtering using KNN classifier

paper_content:

Intrusion detection is one of he important aspects in computer security. Many commercial intrusion detection systems (IDSs) are available and are widely used by organizations. However, most of them suffer from the problem of high false alarm rate, which added heavy workload to security officers who are responsible for handling the alarms. In this paper, we propose a new method to reduce the number of false alarms. We model the normal alarm patterns of IDSs and detect anomaly from incoming alarm streams using k-nearest-neighbor classifier. Preliminary experiments show that our approach successfully reduces up to 93% of false alarms generated by a famous IDS.
---
paper_title: Intrusion detection alarms reduction using root cause analysis and clustering

paper_content:

As soon as the Intrusion Detection System (IDS) detects any suspicious activity, it will generate several alarms referring to as security breaches. Unfortunately, the triggered alarms usually are accompanied with huge number of false positives. In this paper, we use root cause analysis to discover the root causes making the IDS triggers these false alarms; most of these root causes are not attacks. Removing the root causes enhances alarms quality in the future. The root cause instigates the IDS to trigger alarms that almost always have similar features. These similar alarms can be clustered together; consequently, we have designed a new clustering technique to group IDS alarms and to produce clusters. Then, each cluster is modeled by a generalized alarm. The generalized alarms related to root causes are converted (by the security analyst) to filters in order to reduce future alarms' load. The suggested system is a semi-automated system helping the security analyst in specifying the root causes behind these false alarms and in writing accurate filtering rules. The proposed clustering method was verified with three different datasets, and the averaged reduction ratio was about 74% of the total alarms. Application of the new technique to alarms log greatly helps the security analyst in identifying the root causes; and then reduces the alarm load in the future.
---
paper_title: New data mining technique to enhance IDS alarms quality

paper_content:

The intrusion detection systems (IDSs) generate large number of alarms most of which are false positives. Fortunately, there are reasons for triggering alarms where most of these reasons are not attacks. In this paper, a new data mining technique has been developed to group alarms and to produce clusters. Hereafter, each cluster abstracted as a generalized alarm. The generalized alarms related to root causes are converted to filters to reduce future alarms load. The proposed algorithm makes use of nearest neighboring and generalization concepts to cluster alarms. As a clustering algorithm, the proposed algorithm uses a new measure to compute distances between alarms features values. This measure depends on background knowledge of the monitored network, making it robust and meaningful. The new data mining technique was verified with many datasets, and the averaged reduction ratio was about 82% of the total alarms. Application of the new technique to alarms log greatly helps the security analyst in identifying the root causes; and then reduces the alarm load in the future.
---
paper_title: Reducing IDS False Positives Using Incremental Stream Clustering Algorithm

paper_content:

Along with Cryptographic protocols and digital signatures, Intrusion Detection Systems(IDS) are considered to be the last line of defense to secure a network. But the main problem with todays most popular commercial IDSs(Intrusion Detection System) is the generation of huge amount of false positive alerts along with the true positive alerts, which is a cumbersome task for the operator to investigate in order to initiate proper responses. So, there is a great demand to explore this area of research and to find out a feasible solution. ::: ::: In this thesis, we have chosen this problem as our main area of research. We have tested the effectiveness of using the Incremental Stream Clustering Algorithm in order to reduce the number of false alerts from an IDS output. This algorithm was tested with output of one of the most popular network based open source IDS, named Snort, which was configured to playback mood to look for DARPA 1999 network traffic dataset. Our approach was evaluated and compared with K-Nearest Neighbor Algorithm. The result shows that the Incremental Stream Clustering Algorithm reduces (more than 99%) the number of false alarms more than that of K-Nearest Neighbor Algorithm (93%).
---
paper_title: Mining alarm clusters to improve alarm handling efficiency

paper_content:

It is a well-known problem that intrusion detection systems overload their human operators by triggering thousands of alarms per day. As a matter of fact, IBM Research's Zurich Research Laboratory has been asked by one of our service divisions to help them deal with this problem. This paper presents the results of our research, validated thanks to a large set of operational data. We show that alarms should be managed by identifying and resolving their root causes. Alarm clustering is introduced as a method that supports the discovery of root causes. The general alarm clustering problem is proved to be NP-complete, an approximation algorithm is proposed, and experiments are presented.
---
paper_title: A preliminary two-stage alarm correlation and filtering system using SOM neural network and K-means algorithm

paper_content:

Intrusion Detection Systems (IDSs) play a vital role in the overall security infrastructure. Although the IDS has become an essential part of corporate network infrastructure, the art of detecting intrusion is still far from perfect. A significant problem is that of false alarms, as generating a huge volume of such alarms could render the system inefficient. In this paper, we propose a new method to reduce the number of false alarms. We develop a two-stage classification system using a SOM neural network and K-means algorithm to correlate the related alerts and to further classify the alerts into classes of true and false alarms. Preliminary experiments show that our approach effectively reduces all superfluous and noisy alerts, which often contribute to more than 50% of false alarms generated by a common IDS.
---
paper_title: Data Fusion and Cost Minimization for Intrusion Detection

paper_content:

Statistical pattern recognition techniques have recently been shown to provide a finer balance between misdetections and false alarms than the more conventional intrusion detection approaches, namely misuse detection and anomaly detection. A variety of classical machine learning and pattern recognition algorithms has been applied to intrusion detection with varying levels of success. We make two observations about intrusion detection. One is that intrusion detection is significantly more effective by using multiple sources of information in an intelligent way, which is precisely what human experts rely on. Second, different errors in intrusion detection have different costs associated with them-a simplified example being that a false alarm may be more expensive than a misdetection and, hence, the true objective function to be minimized is the cost of errors and not the error rate itself. We present a pattern recognition approach that addresses both of these issues. It utilizes an ensemble of a classifiers approach to intelligently combine information from multiple sources and is explicitly tuned toward minimizing the cost of the errors as opposed to the error rate itself. The information fusion approach dLEARNIN alone is shown to achieve state-of-the-art performances better than those reported in the literature so far, and the cost minimization strategy dCMS further reduces the cost with a significant margin.
---
paper_title: An intrusion detection and alert correlation approach based on revising probabilistic classifiers using expert knowledge

paper_content:

Bayesian networks are important knowledge representation tools for handling uncertain pieces of information. The success of these models is strongly related to their capacity to represent and handle dependence relations. Some forms of Bayesian networks have been successfully applied in many classification tasks. In particular, naive Bayes classifiers have been used for intrusion detection and alerts correlation. This paper analyses the advantage of adding expert knowledge to probabilistic classifiers in the context of intrusion detection and alerts correlation. As examples of probabilistic classifiers, we will consider the well-known Naive Bayes, Tree Augmented Naive Bayes (TAN), Hidden Naive Bayes (HNB) and decision tree classifiers. Our approach can be applied for any classifier where the outcome is a probability distribution over a set of classes (or decisions). In particular, we study how additional expert knowledge such as "it is expected that 80 % of traffic will be normal" can be integrated in classification tasks. Our aim is to revise probabilistic classifiers' outputs in order to fit expert knowledge. Experimental results show that our approach improves existing results on different benchmarks from intrusion detection and alert correlation areas.
---
paper_title: False Alarm Classification Model for Network-Based Intrusion Detection System

paper_content:

Network-based IDS(Intrusion Detection System) gathers network packet data and analyzes them into attack or normal. But they often output a large amount of low-level or incomplete alert information. Such alerts can be unmanageable and also be mixed with false alerts. In this paper we proposed a false alarm classification model to reduce the false alarm rate using classification analysis of data mining techniques. The model was implemented based on associative classification in the domain of DDOS attack. We evaluated the false alarm classifier deployed in front of Snort with Darpa 1998 dataset and verified the reduction of false alarm rate. Our approach is useful to reduce false alerts and to improve the detection rate of network-based intrusion detection systems.
---
paper_title: Using Adaptive Alert Classification to Reduce False Positives in Intrusion Detection

paper_content:

Intrusion Detection Systems (IDSs) are used to monitor computer systems for signs of security violations. Having detected such signs, IDSs trigger alerts to report them. These alerts are presented to a human analyst, who evaluates them and initiates an adequate response.
---
paper_title: Performance enhancement of Intrusion Detection Systems using advances in sensor fusion

paper_content:

Various intrusion detection systems reported in literature have shown distinct preferences for detecting a certain class of attacks with improved accuracy, while performing moderately on the other classes. With the advances in sensor fusion, it has become possible to obtain a more reliable and accurate decision for a wider class of attacks, by combining the decisions of multiple intrusion detection systems. In this paper, an architecture using data-dependent decision fusion is proposed. The method gathers an in-depth understanding about the input traffic and also the behavior of the individual intrusion detection systems by means of a neural network supervised learner unit. This information is used to fine-tune the fusion unit, since the fusion depends on the input feature vector. For illustrative purposes, three intrusion detection systems namely PHAD, ALAD, and Snort have been considered using the DARPA 1999 dataset in order to validate the proposed architecture. The overall performance of the proposed sensor fusion system shows considerable improvement with respect to the performance of individual intrusion detection systems.
---
paper_title: Critical Episode Mining in Intrusion Detection Alerts

paper_content:

One of the most important steps in attack detection using Intrusion Detection Systems (IDSs) is dealing with huge number of alerts that can be either critical single alerts and multi-step attack scenarios or false alerts and non-critical ones. In this paper we try to address the problem of managing alerts via a multi-layer alert correlation and Itering that can identify critical alerts after each step of correlation and Itering. After applying the approach on LL DDoS 1.0 data set, we achieved very good results in terms of critical alert detection rates, running time of approach and its memory usage. Our method could extract all of critical and multi-step attacks in LL DDoS 1.0 data set while we had almost 90% reduction in number of alerts.
---
paper_title: Applying Data Fusion in Collaborative Alerts Correlation

paper_content:

Due to various network intrusions, network security has always been a main concern of the network administrator. However, nowadays traditional security tools like IDSs, firewalls etc cannot play the roles of effective defense mechanisms. Instead, they only generate elementary alerts to form alert flooding and they often have high false alerts rates. Moreover due to their weak collaboration-awareness, they cannot detect large distributed attacks such as a DDoS attack. In this paper, we present an efficient and effective model for collaborative alerts analyzing. Our system enhances the alert verification using assets? contextual information. By applying alert fusion and using a precisely defined knowledge base in the correlation phase, it also provides a method to get general and synthetic alerts from the large volume of elementary alerts. Moreover, this system is able to reconstruct the attack scenarios for multi-step attacks. Experiments show the system can effectively distinguish false positives, detect and predicate large-scale attacks in their early stage.
---
paper_title: An Operational Framework for Alert Correlation using a Novel Clustering Approach

paper_content:

Intrusion Detection System (IDS) is a well known security feature and widely implemented among practitioners. However, since the creation of IDS the enormous number of alerts generated by the detection sensors has always been a setback in the implementation environment. Moreover due to this obtrusive predicament, two other problems have emerged which are the difficulty in processing the alerts accurately and also the decrease in performance rate in terms of time and memory capacity while processing these alerts. Thus, based on the specified problems, the purpose of our overall research is to construct a holistic solution that is able to reduce the number of alerts to be processed and at the same time to produce a high quality attack scenarios that are meaningful to the administrators in a timely manner. In this paper we will present our proposed framework together with the result of our novel clustering method, architectured solely with the intention of reducing the amount of alerts generated by IDS. The clustering method was tested against two dataset; a globally used dataset, DARPA and a live dataset from a cyber attack monitoring unit that uses SNORT engine to capture the alerts. The result obtained from the experiment is very promising; the clustering algorithm was able to reduce about 86.9% of the alerts used in the experiment. From the result we are able to highlight the contribution to practitioners in an actual working environment.
---
paper_title: Comprehensive approach to intrusion detection alert correlation

paper_content:

Alert correlation is a process that analyzes the alerts produced by one or more intrusion detection systems and provides a more succinct and high-level view of occurring or attempted intrusions. Even though the correlation process is often presented as a single step, the analysis is actually carried out by a number of components, each of which has a specific goal. Unfortunately, most approaches to correlation concentrate on just a few components of the process, providing formalisms and techniques that address only specific correlation issues. This paper presents a general correlation model that includes a comprehensive set of components and a framework based on this model. A tool using the framework has been applied to a number of well-known intrusion detection data sets to identify how each component contributes to the overall goals of correlation. The results of these experiments show that the correlation components are effective in achieving alert reduction and abstraction. They also show that the effectiveness of a component depends heavily on the nature of the data set analyzed.
---
paper_title: The Intrusion Detection Message Exchange Format (IDMEF)

paper_content:

The purpose of the Intrusion Detection Message Exchange Format (IDMEF) ::: is to define data formats and exchange procedures for sharing ::: information of interest to intrusion detection and response systems ::: and to the management systems that may need to interact with them. ::: This document describes a data model to represent information exported ::: by intrusion detection systems and explains the rationale for using ::: this model. An implementation of the data model in the Extensible ::: Markup Language (XML) is presented, an XML Document Type Definition is ::: developed, and examples are provided. This memo defines an ::: Experimental Protocol for the Internet community.
---
paper_title: IDS alerts correlation using grammar-based approach

paper_content:

Intrusion Detection System (IDS) is a security technology that attempts to identify intrusions. Defending against multi-step intrusions which prepare for each other is a challenging task. In this paper, we propose a novel approach to alert post-processing and correlation, the Alerts Parser. Different from most other alert correlation methods, our approach treats the alerts as tokens and uses modified version of the LR parser to generate parse trees representing the scenarii in the alerts. An Attribute Context-Free Grammar (ACF-grammar) is used for representing the multi-step attacks. Attack scenarii information and prerequisites/consequences knowledge are included together in the ACF-grammar enhancing the correlation results. The modified LR parser depends on these ACF-grammars to generate parse trees. The experiments were performed on two different sets of network traffic traces, using different open-source and commercial IDS sensors. The discovered scenarii are represented by Correlation Graphs (CGs). The experimental results show that Alerts Parser can work in parallel, effectively correlate related alerts with low false correlation rate, uncover the attack strategies, and generate concise CGs.
---
paper_title: Alert correlation in a cooperative intrusion detection framework

paper_content:

This paper presents the work we have done within the MIRADOR project to design CRIM, a cooperative module for intrusion detection systems (IDS). This module implements functions to manage, cluster, merge and correlate alerts. The clustering and merging functions recognize alerts that correspond to the same occurrence of an attack and create a new alert that merge data contained in these various alerts. Experiments show that these functions significantly reduce the number of alerts. However, we also observe that alerts we obtain are still too elementary to be managed by a security administrator. The purpose of the correlation function is thus to generate global and synthetic alerts. This paper focuses on the approach we suggest to design this function.
---
paper_title: Modeling network intrusion detection alerts for correlation

paper_content:

Signature-based network intrusion-detection systems (NIDSs) often report a massive number of simple alerts of low-level security-related events. Many of these alerts are logically involved in a single multi-stage intrusion incident and a security officer often wants to analyze the complete incident instead of each individual simple alert. This paper proposes a well-structured model that abstracts the logical relation between the alerts in order to support automatic correlation of those alerts involved in the same intrusion. The basic building block of the model is a logical formula called a capability. We use capability to abstract consistently and precisely all levels of accesses obtained by the attacker in each step of a multistage intrusion. We then derive inference rules to define logical relations between different capabilities. Based on the model and the inference rules, we have developed several novel alert correlation algorithms and implemented a prototype alert correlator. The experimental results of the correlator using several intrusion datasets demonstrate that the approach is effective in both alert fusion and alert correlation and has the ability to correlate alerts of complex multistage intrusions. In several instances, the alert correlator successfully correlated more than two thousand Snort alerts involved in massive scanning incidents. It also helped us find two multistage intrusions that were missed in auditing by the security officers.
---
paper_title: Modeling Multistep Cyber Attacks for Scenario Recognition

paper_content:

Efforts toward automated detection and identification of multistep cyber attack scenarios would benefit significantly from a methodology and language for modeling such scenarios. The Correlated Attack Modeling Language (CAML) uses a modular approach, where a module represents an inference step and modules can be linked together to detect multistep scenarios. CAML is accompanied by a library of predicates, which functions as a vocabulary to describe the properties of system states and events. The concept of attack patterns is introduced to facilitate reuse of generic modules in the attack modeling process. CAML is used in a prototype implementation of a scenario recognition engine that consumes first-level security alerts in real time and produces reports that identify multistep attack scenarios discovered in the alert stream.
---
paper_title: A Novel Framework for Alert Correlation and Understanding

paper_content:

We propose a novel framework named Hidden Colored Petri-Net for Alert Correlation and Understanding (HCPN-ACU) in intrusion detection system. This model is based upon the premise that intrusion detection may be viewed as an inference problem – in other words, we seek to show that system misusers are carrying out a sequence of steps to violate system security policies in some way, with earlier steps preparing for the later ones. In contrast with prior arts, we separate actions from observations and assume that the attacker’s actions themselves are unknown, but the attacker’s behavior may result in alerts. These alerts are then used to infer the attacker’s actions. We evaluate the model with DARPA evaluation database. We conclude that HCPN-ACU can conduct alert fusion and intention recognition at the same time, reduce false positives and negatives, and provide better understanding of the intrusion progress by introducing confidence scores.
---
paper_title: Analyzing Intensive Intrusion Alerts via Correlation

paper_content:

Traditional intrusion detection systems (IDSs) focus on low-level attacks or anomalies, and raise alerts independently, though there may be logical connections between them. In situations where there are intensive intrusions, not only will actual alerts be mixed with false alerts, but the amount of alerts will also become unmanageable. As a result, it is difficult for human users or intrusion response systems to understand the alerts and take appropriate actions. Several complementary alert correlation methods have been proposed to address this problem. As one of these methods, we have developed a framework to correlate intrusion alerts using prerequisites of intrusions. In this paper, we continue this work to study the feasibility of this method in analyzing real-world, intensive intrusions. In particular, we develop three utilities (called adjustable graph reduction, focused analysis, and graph decomposition) to facilitate the analysis of large sets of correlated alerts. We study the effectiveness of the alert correlation method and these utilities through a case study with the network traffic captured at the DEF CON 8 Capture the Flag (CTF) event. Our results show that these utilities can simplify the analysis of large amounts of alerts, and also reveals several attack strategies that were repeatedly used in the DEF CON 8 CTF event.
---
paper_title: Techniques and tools for analyzing intrusion alerts

paper_content:

Traditional intrusion detection systems (IDSs) focus on low-level attacks or anomalies, and raise alerts independently, though there may be logical connections between them. In situations where there are intensive attacks, not only will actual alerts be mixed with false alerts, but the amount of alerts will also become unmanageable. As a result, it is difficult for human users or intrusion response systems to understand the alerts and take appropriate actions. This paper presents a sequence of techniques to address this issue. The first technique constructs attack scenarios by correlating alerts on the basis of prerequisites and consequences of attacks. Intuitively, the prerequisite of an attack is the necessary condition for the attack to be successful, while the consequence of an attack is the possible outcome of the attack. Based on the prerequisites and consequences of different types of attacks, the proposed method correlates alerts by (partially) matching the consequences of some prior alerts with the prerequisites of some later ones. Moreover, to handle large collections of alerts, this paper presents a set of interactive analysis utilities aimed at facilitating the investigation of large sets of intrusion alerts. This paper also presents the development of a toolkit named TIAA, which provides system support for interactive intrusion analysis. This paper finally reports the experiments conducted to validate the proposed techniques with the 2000 DARPA intrusion detection scenario-specific datasets, and the data collected at the DEFCON 8 Capture the Flag event.
---
paper_title: LAMBDA: A Language to Model a Database for Detection of Attacks

paper_content:

This article presents an attack description language. This language is based on logic and uses a declarative approach. In the language, the conditions and effects of an attack are described with logical formulas related to the state of the target computer system. The various steps of the attack process are associated to events, which may be combined using specific algebraic operators. These elements provide a description of the attack from the point of view of the attacker. They are complemented with additional elements corresponding to the point of view of intrusion detection systems and audit programs. These detection and verification aspects provide the language user with means to tailor the description of the attack to the needs of a specific intrusion detection system or a specific environment.
---
paper_title: A requires/provides model for computer attacks

paper_content:

Computer attacks are typically described in terms of a single exploited vulnerability or as a signature composed of a specific sequence of events. These approaches lack the ability to characterize complex scenarios or to generalize to unknown attacks. Rather than think of attacks as a series of events, we view attacks as a set of capabilities that provide support for abstract attack concepts that in turn provide new capabilities to support other concepts. This paper describes a flexible extensible model for computer attacks, a language for specifying the model, and how it can be used in security applications such as vulnerability analysis, intrusion detection and attack generation
---
paper_title: An Improved Framework for Intrusion Alert Correlation

paper_content:

Alert correlation analyzes the alerts from one or more collaborative Intrusion Detection Systems (IDSs) to produce a concise overview of security-related activity on the network. The process consists of multiple components, each responsible for a different aspect of the overall correlation goal. The sequence order of the correlation components affects the correlation process performance. The total time needed for the whole process depends on the number of processed alerts in each component. This paper proposes a new correlation framework based on a model that reduces the number of processed alerts as early as possible by discarding the irrelevant and false alerts in the first phases. A new component is added to deal with the unrelated alerts. A modified algorithm for fusing the alerts is also proposed. The intruders’ intention is grouped into attack scenarios and thus used to detect future attacks. The contribution of this paper includes an enhanced new framework for alert correlation, the implementation of the alert correlator model based on the framework, and the evaluation of the model using the DARPA 2000 intrusion detection scenario specific datasets. The experimental results show that the correlation model is effective in achieving alert reduction and abstraction. The performance is improved after the attention is focused on correlating higher severity alerts.
---
paper_title: TRINETR: an intrusion detection alert management systems

paper_content:

In response to the daunting threats of cyber attacks, a promising approach is computer and network forensics. Intrusion detection system is an indispensable part of computer and network forensics. It is deployed to monitor network and host activities including dataflows and information accesses etc. But current intrusion detection products presents many flaws including alert flooding, too many false alerts and isolated alerts etc. We describe an ongoing project to develop an intrusion alert management system $TRINETR. We present a collaborative architecture design for multiple intrusion detection systems to work together to detect real-time network intrusions. The architecture is composed of three parts: alert aggregation, knowledge-based alert evaluation and alert correlation. The architecture is aimed at reducing the alert overload by aggregating alerts from multiple sensors to generate condensed views, reducing false positives by integrating network and host system information into alert evaluation process and correlating events based on logical relations to generate global and synthesized alert report. The first two parts of the architecture have been implemented and the implementation results are presented.
---
paper_title: M2d2: A formal data model for ids alert correlation

paper_content:

At present, alert correlation techniques do not make full use of the information that is available. We propose a data model for IDS alert correlation called M2D2. It supplies four information types: information related to the characteristics of the monitored information system, information about the vulnerabilities, information about the security tools used for the monitoring, and information about the events observed. M2D2 is formally defined. As far as we know, no other formal model includes the vulnerability and alert parts of M2D2. Three examples of correlations are given. They are rigorously specified using the formal definition of M2D2. As opposed to already published correlation methods, these examples use more than the events generated by security tools; they make use of many concepts formalized in M2D2.
---
paper_title: Information modeling for intrusion report aggregation

paper_content:

The paper describes the SCYLLARUS approach to fusing reports from multiple intrusion detection systems (ID-Ses) to provide an overall approach to intrusion situation awareness. The overall view provided by SCYLLARUS centers around the site's security goals, aggregating large numbers of individual IDS reports based on their impact. The overall view reduces information overload by aggregating multiple IDS reports in a rep-down view; and by reducing false positives by weighing evidence provided by multiple ID-Ses and other information sources. Unlike previous efforts in this area, SCYLLARUS is centered around its intrusion reference model (IRM). The SCYLLARUS IRM contains both dynamic and static (configuration) information. A network entity/relationship database (NERD), providing information about the site's hardware and software; a security goal database, describing the site's objectives and security policy; and an event dictionary, describing important events, both intrusions and benign; comprise the static portion of the IRM. The set of IDS reports; the events SCYLLARUS hypothesizes to explain them; and the resulting judgment of the state of site security goals comprise the dynamic part of the IRM.
---
paper_title: Comprehensive approach to intrusion detection alert correlation

paper_content:

Alert correlation is a process that analyzes the alerts produced by one or more intrusion detection systems and provides a more succinct and high-level view of occurring or attempted intrusions. Even though the correlation process is often presented as a single step, the analysis is actually carried out by a number of components, each of which has a specific goal. Unfortunately, most approaches to correlation concentrate on just a few components of the process, providing formalisms and techniques that address only specific correlation issues. This paper presents a general correlation model that includes a comprehensive set of components and a framework based on this model. A tool using the framework has been applied to a number of well-known intrusion detection data sets to identify how each component contributes to the overall goals of correlation. The results of these experiments show that the correlation components are effective in achieving alert reduction and abstraction. They also show that the effectiveness of a component depends heavily on the nature of the data set analyzed.
---
paper_title: NetSTAT: a network-based intrusion detection approach

paper_content:

Network-based attacks have become common and sophisticated. For this reason, intrusion detection systems are now shifting their focus from the hosts and their operating systems to the network itself. Network-based intrusion detection is challenging because network auditing produces large amounts of data, and different events related to a single intrusion may be visible in different places on the network. This paper presents NetSTAT, a new approach to network intrusion detection. By using a formal model of both the network and the attacks, NetSTAT is able to determine which network events have to be monitored and where they can be monitored.
---
paper_title: An approach to sensor correlation

paper_content:

We present an approach to intrusion detection (ID) sensor correlation that considers the problem in three phases: event aggregation, sensor coupling, and meta alert fusion. The approach is well suited to probabilistically based sensors such as EMERALD eBayes. We demonstrate the efficacy of the EMERALD alert thread mechanism, the sensor coupling in eBayes, and a prototype alert fusion capability towards achieving significant functionality in the field of ID sensor correlation.
---
paper_title: Analysis of Credential Stealing Attacks in an Open Networked Environment

paper_content:

This paper analyses the forensic data on credential stealing incidents over a period of 5 years across 5000 machines monitored at the National Center for Supercomputing Applications at the University of Illinois. The analysis conducted is the first attempt in an open operational environment (i) to evaluate the intricacies of carrying out SSH-based credential stealing attacks, (ii) to highlight and quantify key characteristics of such attacks, and (iii) to provide the system level characterization of such incidents in terms of distribution of alerts and incident consequences
---
paper_title: Alarm reduction and correlation in defence of IP networks

paper_content:

Society's critical infrastructures are increasingly dependent on IP networks. Intrusion detection and tolerance within data networks is therefore imperative for dependability in other domains such as telecommunications and future energy management networks. Today's data networks are protected by human operators who are overwhelmed by the massive information overload through false alarm rates of the protection mechanisms. This paper studies the role of alarm reduction and correlation in supporting the security administrator in an enterprise network. We present an architecture that incorporates intrusion detection systems as sensors, and provides improved alarm data to the human operator or to automated actuators. Alarm reduction and correlation via static and adaptive filtering, normalisation, and aggregation is demonstrated on the output from three sensors (Snort, Samhain and Syslog) used in a telecom test network.
---
paper_title: Aggregation and Correlation of Intrusion-Detection Alerts

paper_content:

This paper describes an aggregation and correlation algorithm used in the design and implementation of an intrusion-detection console built on top of the Tivoli Enterprise Console (TEC). The aggregation and correlation algorithm aims at acquiring intrusion-detection alerts and relating them together to expose a more condensed view of the security issues raised by intrusion-detection systems.
---
paper_title: Analysis of security data from a large computing organization

paper_content:

This paper presents an in-depth study of the forensic data on security incidents that have occurred over a period of 5 years at the National Center for Supercomputing Applications at the University of Illinois. The proposed methodology combines automated analysis of data from security monitors and system logs with human expertise to extract and process relevant data in order to: (i) determine the progression of an attack, (ii) establish incident categories and characterize their severity, (iii) associate alerts with incidents, and (iv) identify incidents missed by the monitoring tools and examine the reasons for the escapes. The analysis conducted provides the basis for incident modeling and design of new techniques for security monitoring.
---
paper_title: On the use of different statistical tests for alert correlation – Short Paper

paper_content:

In this paper we analyze the use of different types of statistical tests for the correlation of anomaly detection alerts. We show that the Granger Causality Test, one of the few proposals that can be extended to the anomaly detection domain, strongly depends on good choices of a parameter which proves to be both sensitive and difficult to estimate. We propose a different approach based on a set of simpler statistical tests, and we prove that our criteria work well on a simplified correlation task, without requiring complex configuration parameters.
---
paper_title: Discovering novel attack strategies from INFOSEC alerts

paper_content:

Correlating security alerts and discovering attack strategies are important and challenging tasks for security analysts. Recently, there have been several proposed techniques to analyze attack scenarios from security alerts. However, most of these approaches depend on a priori and hard-coded domain knowledge that lead to their limited capabilities of detecting new attack strategies. In this paper, we propose an approach to discover novel attack strategies. Our approach includes two complementary correlation mechanisms based on two hypotheses of attack step relationship. The first hypothesis is that attack steps are directly related because an earlier attack enables or positively affects the later one. For this type of attack relationship, we develop a Bayesian-based correlation engine to correlate attack steps based on security states of systems and networks. The second hypothesis is that for some related attack steps, even though they do not have obvious and direct relationship in terms of security and performance measures, they still have temporal and statistical patterns. For this category of relationship, we apply time series and statistical analysis to correlate attack steps. The security analysts are presented with aggregated information on attack strategies from these two correlation engines. We evaluate our approach using DARPA’s Grand Challenge Problem (GCP) data sets. The results show that our approach can discover novel attack strategies and provide a quantitative analysis of attack scenarios.
---
paper_title: An online adaptive approach to alert correlation

paper_content:

The current intrusion detection systems (IDSs) generate a tremendous number of intrusion alerts. In practice, managing and analyzing this large number of low-level alerts is one of the most challenging tasks for a system administrator. In this context alert correlation techniques aiming to provide a succinct and high-level view of attacks gained a lot of interest. Although, a variety of methods were proposed, the majority of them address the alert correlation in the off-line setting. In this work, we focus on the online approach to alert correlation. Specifically, we propose a fully automated adaptive approach for online correlation of intrusion alerts in two stages. In the first online stage, we employ a Bayesian network to automatically extract information about the constraints and causal relationships among alerts. Based on the extracted information, we reconstruct attack scenarios on-the-fly providing network administrator with the current network view and predicting the next potential steps of the attacker. Our approach is illustrated using both the well known DARPA 2000 data set and the live traffic data collected from a Honeynet network.
---
paper_title: Processing intrusion detection alert aggregates with time series modeling

paper_content:

The main use of intrusion detection systems (IDS) is to detect attacks against information systems and networks. Normal use of the network and its functioning can also be monitored with an IDS. It can be used to control, for example, the use of management and signaling protocols, or the network traffic related to some less critical aspects of system policies. These complementary usages can generate large numbers of alerts, but still, in operational environment, the collection of such data may be mandated by the security policy. Processing this type of alerts presents a different problem than correlating alerts directly related to attacks or filtering incorrectly issued alerts. We aggregate individual alerts to alert flows, and then process the flows instead of individual alerts for two reasons. First, this is necessary to cope with the large quantity of alerts - a common problem among all alert correlation approaches. Second, individual alert's relevancy is often indeterminable, but irrelevant alerts and interesting phenomena can be identified at the flow level. This is the particularity of the alerts created by the complementary uses of IDSes. Flows consisting of alerts related to normal system behavior can contain strong regularities. We propose to model these regularities using non-stationary autoregressive models. Once modeled, the regularities can be filtered out to relieve the security operator from manual analysis of true, but low impact alerts. We present experimental results using these models to process voluminous alert flows from an operational network.
---
paper_title: IDS interoperability and correlation using IDMEF and commodity systems

paper_content:

Over the past decade Intrusion Detection Systems (IDS) have been steadily improving their efficiency and effectiveness in detecting attacks. This is particularly true with signature-based IDS due to progress in attack analysis and attack signature specification. At the same time system complexity, overall numbers of bugs and security vulnerabilities have increased. This has led to the recognition that in order to operate over the entire attack space, multiple IDS must be used, which need to interoperate with one another, and possibly also with other components of system security. This paper describes an experiment in IDS interoperation using the Intrusion Detection Message Exchange Format for the purpose of correlation analysis and in order to identify and address the problems associated with the effective use and management of multiple IDS. A study of the process of intrusion analysis demonstrates the benefits of multi-IDS interoperation and cooperation, as well as the significant benefits provided by alert analysis using a central relational database.
---
paper_title: Alert Fusion for a Computer Host Based Intrusion Detection System

paper_content:

Intrusions impose tremendous threats to today's computer hosts. Intrusions using security breaches to achieve unauthorized access or misuse of critical information can have catastrophic consequences. To protect computer hosts from the increasing threat of intrusion, various kinds of intrusion detection systems (IDSs) have been developed. The main disadvantages of current IDSs are a high false detection rate and the lack of post-intrusion decision support capability. To minimize these drawbacks, we propose an event-driven intrusion detection architecture which integrates subject-verb-object (SVO) multi-point monitors and an impact analysis engine. Alert fusion and verification models are implemented to provide more reasonable intrusion information from incomplete, inconsistent or imprecise alerts acquired by SVO monitors. DEVS formalism is used to describe the model based design approach. Finally we use the DEVS-JAVA simulation tool to show the feasibility of the proposed system
---
paper_title: Optimal IDS Sensor Placement and Alert Prioritization Using Attack Graphs

paper_content:

We optimally place intrusion detection system (IDS) sensors and prioritize IDS alerts using attack graph analysis. We begin by predicting all possible ways of penetrating a network to reach critical assets. The set of all such paths through the network constitutes an attack graph, which we aggregate according to underlying network regularities, reducing the complexity of analysis. We then place IDS sensors to cover the attack graph, using the fewest number of sensors. This minimizes the cost of sensors, including effort of deploying, configuring, and maintaining them, while maintaining complete coverage of potential attack paths. The sensor-placement problem we pose is an instance of the NP-hard minimum set cover problem. We solve this problem through an efficient greedy algorithm, which works well in practice. Once sensors are deployed and alerts are raised, our predictive attack graph allows us to prioritize alerts based on attack graph distance to critical assets.
---
paper_title: Time-efficient and cost-effective network hardening using attack graphs

paper_content:

Attack graph analysis has been established as a powerful tool for analyzing network vulnerability. However, previous approaches to network hardening look for exact solutions and thus do not scale. Further, hardening elements have been treated independently, which is inappropriate for real environments. For example, the cost for patching many systems may be nearly the same as for patching a single one. Or patching a vulnerability may have the same effect as blocking traffic with a firewall, while blocking a port may deny legitimate service. By failing to account for such hardening interdependencies, the resulting recommendations can be unrealistic and far from optimal. Instead, we formalize the notion of hardening strategy in terms of allowable actions, and define a cost model that takes into account the impact of interdependent hardening actions. We also introduce a near-optimal approximation algorithm that scales linearly with the size of the graphs, which we validate experimentally.
---
paper_title: Building Attack Scenarios through Integration of Complementary Alert Correlation Method

paper_content:

Several alert correlation methods were proposed in the past several years to construct high-level attack scenarios from low-level intrusion alerts reported by intrusion detection systems (IDSs). These correlation methods have different strengths and limitations; none of them clearly dominate the others. However, all of these methods depend heavily on the underlying IDSs, and perform poorly when the IDSs miss critical attacks. In order to improve the performance of intrusion alert correlation and reduce the impact of missed attacks, this paper presents a series of techniques to integrate two complementary types of alert correlation methods: (1) those based on the similarity between alert attributes, and (2) those based on prerequisites and consequences of attacks. In particular, this paper presents techniques to hypothesize and reason about attacks possibly missed by IDSs based on the indirect causal relationship between intrusion alerts and the constraints they must satisfy. This paper also discusses additional techniques to validate the hypothesized attacks through raw audit data and to consolidate the hypothesized attacks to generate concise attack scenarios. The experimental results in this paper demonstrate the potential of these techniques in building high-level attack scenarios and reasoning about possibly missed attacks.
---
paper_title: A new alert correlation algorithm based on attack graph

paper_content:

Intrusion Detection Systems (IDS) are widely deployed in computer networks. As modern attacks are getting more sophisticated and the number of sensors and network nodes grows, the problem of false positives and alert analysis becomes more difficult to solve. Alert correlation was proposed to analyze alerts and to decrease false positives. Knowledge about the target system or environment is usually necessary for efficient alert correlation. For representing the environment information as well as potential exploits, the existing vulnerabilities and their Attack Graph (AG) is used. It is useful for networks to generate an AG and to organize certain vulnerabilities in a reasonable way. In this paper, we design a correlation algorithm based on AGs that is capable of detecting multiple attack scenarios for forensic analysis. It can be parameterized to adjust the robustness and accuracy. A formal model of the algorithm is presented and an implementation is tested to analyze the different parameters on a real set of alerts from a local network.
---
paper_title: An Intrinsic Graphical Signature Based on Alert Correlation Analysis for Intrusion Detection

paper_content:

We propose a graphical signature for intrusion detection given alert sequences. By correlating alerts with their temporal proximity, we build a probabilistic graph-based model to describe a group of alerts that form an attack or normal behavior. Using the models, we design a pairwise measure based on manifold learning to measure the dissimilarities between different groups of alerts. A large dissimilarity implies different behaviors between the two groups of alerts. Such measure can therefore be combined with regular classification methods for intrusion detection. We evaluate our framework mainly on Acer 2007, a private dataset gathered from a well-known Security Operation Center in Taiwan. The performance on the real data suggests that the proposed method can achieve high detection accuracy. Moreover, the graphical structures and the representation from manifold learning naturally provide the visualized result suitable for further analysis from domain experts.
---
paper_title: Alert Verification Determining the Success of Intrusion Attempts

paper_content:

Recently, intrusion detection systems (IDSs) have been increasingly brought to task for failing to meet the expectations that researchers and vendors were raising. Promises that IDSs would be capable of reliably identifying malicious activity never turned into reality. While virus scanners and firewalls have visible benefits and remain virtually unnoticed during normal operation, intrusion detection systems are known for producing a large number of alerts that are either not related to malicious activity (false positives) or not representative of a successful attack (non-relevant positives). Although tuning and proper configuration may eliminate the most obvious spurious alerts, the problem of the vast imbalance between actual and false or non-relevant alerts remains.
---
paper_title: Using Alert Verification to Identify Successful Intrusion Attempts

paper_content:

Intrusion detection systems monitor protected networks and attempt to identify evidence of malicious activity. When an attack is detected, an alert is produced, and, possibly, a countermeasure is executed. A perfect intrusion detection system would be able to identify all the attacks without raising any false alarms. In addition, a countermeasure would be executed only when an attack is actually successful. Unfortunately false alarms are commonplace in intrusion detection systems, and perfectly benign events are interpreted as malicious. In addition, non-relevant alerts are also common. These are alerts associated with attacks that were not successful. Such alerts should be tagged appropriately so that their priority can be lowered. The process of identifying alerts associated with successful attacks is called alert verification. This paper describes the different issues involved in alert verification and presents a tool that performs real-time verification of attacks detected by an intrusion detection system. The experimental evaluation of the tool shows that verification can dramatically reduce both false and non-relevant alerts.
---
paper_title: Verify results of network intrusion alerts using lightweight protocol analysis

paper_content:

We propose a method to verify the result of attacks detected by signature-based network intrusion detection systems using lightweight protocol analysis. The observation is that network protocols often have short meaningful status codes saved at the beginning of server responses upon client requests. A successful intrusion that alters the behavior of a network application server often results in an unexpected server response, which does not contain the valid protocol status code. This can be used to verify the result of the intrusion attempt. We then extend this method to verify the result of attacks that still generate valid protocol status code in the server responses. We evaluate this approach by augmenting Snort signatures and testing on real world data. We show that some simple changes to Snort signatures can effectively verify the result of attacks against the application servers, thus significantly improve the quality of alerts
---
paper_title: ATLANTIDES: An Architecture for Alert Verification in Network Intrusion Detection Systems

paper_content:

We present an architecture designed for alert verification (i.e., to reduce false positives) in network intrusion-detection systems. Our technique is based on a systematic (and automatic) anomaly-based analysis of the system output, which provides useful context information regarding the network services. The false positives raised by the NIDS analyzing the incoming traffic (which can be either signature- or anomaly-based) are reduced by correlating them with the output anomalies. We designed our architecture for TCP-based network services which have a client/server architecture (such as HTTP). Benchmarks show a substantial reduction of false positives between 50% and 100%.
---
paper_title: Alert verification evasion through server response forging

paper_content:

Intrusion Detection Systems (IDSs) are necessary components in the defense of any computer network. Network administrators rely on IDSs to detect attacks, but ultimately it is their responsibility to investigate IDS alerts and determine the damage done. With the number of alerts increasing, IDS analysts have turned to automated methods to help with alert verification. This research investigates this next step of the intrusion detection process. Some alert verification mechanisms attempt to identify successful intrusion attempts based on server responses and protocol analysis. This research examines the server responses generated by four different exploits across four different Linux distributions. Next, three techniques capable of forging server responses on Linux operating systems are developed and implemented. This research shows that these new alert verification evasion methods can make attacks appear unsuccessful even though the exploitation occurs. This type of attack ignores detection and tries to evade the verification process.
---
paper_title: Monitoring IDS Background Noise Using EWMA Control Charts and Alert Information

paper_content:

Intrusion detection systems typically create large amounts of alerts, processing of which is a time consuming task for the user. This paper describes an application of exponentially weighted moving average (EWMA) control charts used to help the operator in alert processing. Depending on his objectives, some alerts are individually insignificant, but when aggregated they can provide important information on the monitored system’s state. Thus it is not always the best solution to discard those alerts, for instance, by means of filtering, correlation, or by simply removing the signature. We deploy a widely used EWMA control chart for extracting trends and highlighting anomalies from alert information provided by sensors performing pattern matching. The aim is to make output of verbose signatures more tolerable for the operator and yet allow him to obtain the useful information available. The applied method is described and experimentation along its results with real world data are presented. A test metric is proposed to evaluate the results.
---
paper_title: Online Risk Assessment of Intrusion Scenarios Using D-S Evidence Theory

paper_content:

In the paper, an online risk assessment model based on D-S evidence theory is presented. The model can quantitate the risk caused by an intrusion scenario in real time and provide an objective evaluation of the target security state. The results of the online risk assessment show a clear and concise picture of both the intrusion progress and the target security state. The model makes full use of available information from both IDS alerts and protected targets. As a result, it can deal with uncertainties and subjectiveness very well in its evaluation process. In IDAM&IRS, the model serves as the foundation for intrusion response decision-making.
---
paper_title: Intrusion detection alert verification based on multi-level fuzzy comprehensive evaluation

paper_content:

Alert verification is a process which compares the information referred by an alert with the configuration and topology information of its target system in order to determine if the alert is relevant to its target system. It can reduce false positive alerts and irrelevant alerts. The paper presents an alert verification approach based on multi-level fuzzy comprehensive evaluation. It is effective in achieving false alert and irrelevant alerts reduction, which have been proved by our experiments. The algorithm can deal with the uncertainties better than other alert verification approaches. The relevance score vectors obtained from the algorithm facilitate the formulation of fine and flexible security policies, and further alert processing.
---
paper_title: Alert prioritization in Intrusion Detection Systems

paper_content:

Intrusion Detection Systems (IDSs) are designed to monitor user and/or network activity and generate alerts whenever abnormal activities are detected. The number of these alerts can be very large; making the task of security analysts difficult to manage. Furthermore, IDS alert management techniques, such as clustering and correlation, suffer from involving unrelated alerts in their processes and consequently provide imprecise results. In this paper, we propose a fuzzy-logic based technique for scoring and prioritizing alerts generated by an IDS(1). In addition, we present an alert rescoring technique that leads to a further reduction of the number of alerts. The approach is validated using the 2000 DARPA intrusion detection scenario specific datasets and comparative results between the Snort IDS alert scoring and our scoring and prioritization scheme are presented.
---
paper_title: A mission-impact-based approach to INFOSEC alarm correlation

paper_content:

We describe a mission-impact-based approach to the analysis of security alerts produced by spatially distributed heterogeneous information security (INFOSEC) devices, such as firewalls, intrusion detection systems, authentication services, and antivirus software. The intent of this work is to deliver an automated capability to reduce the time and cost of managing multiple INFOSEC devices through a strategy of topology analysis, alert prioritization, and common attribute-based alert aggregation. Our efforts to date have led to the development of a prototype system called the EMERALD Mission Impact Intrusion Report Correlation System, or M-Correlator. M-Correlator is intended to provide analysts (at all experience levels) a powerful capability to automatically fuse together and isolate those INFOSEC alerts that represent the greatest threat to the health and security of their networks.
---
paper_title: Reducing false positives in intrusion detection systems

paper_content:

A post-processing filter is proposed to reduce false positives in network-based intrusion detection systems. The filter comprises three components, each one of which is based upon statistical properties of the input alert set. Special characteristics of alerts corresponding to true attacks are exploited. These alerts may be observed in batches, which contain similarities in the source or destination IPs, or they may produce abnormalities in the distribution of alerts of the same signature. False alerts can be recognized by the frequency with which their signature triggers false positives. The filter architecture and design are discussed. Evaluation results performed using the DARPA 1999 dataset indicate that the proposed approach can significantly reduce the number and percentage of false positives produced by Snort(C) (Roesch, 1999). Our filter limited false positives by a percentage up to 75%.
---
paper_title: Snort - Lightweight Intrusion Detection for Networks

paper_content:

Network intrusion detection systems (NIDS) are an important part of any network security architecture. They provide a layer of defense which monitors network traffic for predefined suspicious activity or patterns, and alert system administrators when potential hostile traffic is detected. Commercial NIDS have many differences, but Information Systems departments must face the commonalities that they share such as significant system footprint, complex deployment and high monetary cost. Snort was designed to address these issues.
---
paper_title: Bro: A System for Detecting Network Intruders in Real-Time

paper_content:

Abstract We describe Bro, a stand-alone system for detecting network intruders in real-time by passively monitoring a network link over which the intruder's traffic transits. We give an overview of the system's design, which emphasizes high-speed (FDDI-rate) monitoring, real-time notification, clear separation between mechanism and policy, and extensibility. To achieve these ends, Bro is divided into an `event engine' that reduces a kernel-filtered network traffic stream into a series of higher-level events, and a `policy script interpreter' that interprets event handlers written in a specialized language used to express a site's security policy. Event handlers can update state information, synthesize new events, record information to disk, and generate real-time notifications via syslog. We also discuss a number of attacks that attempt to subvert passive monitoring systems and defenses against these, and give particulars of how Bro analyzes the six applications integrated into it so far: Finger, FTP, Portmapper, Ident, Telnet and Rlogin. The system is publicly available in source code form.
---
paper_title: VisFlowConnect: netflow visualizations of link relationships for security situational awareness

paper_content:

We present a visualization design to enhance the ability of an administrator to detect and investigate anomalous traffic between a local network and external domains. Central to the design is a parallel axes view which displays NetFlow records as links between two machines or domains while employing a variety of visual cues to assist the user. We describe several filtering options that can be employed to hide uninteresting or innocuous traffic such that the user can focus his or her attention on the more unusual network flows. This design is implemented in the form of VisFlowConnect, a prototype application which we used to study the effectiveness of our visualization approach. Using VisFlowConnect, we were able to discover a variety of interesting network traffic patterns. Some of these were harmless, normal behavior, but some were malicious attacks against machines on the network.
---
paper_title: Correlation between NetFlow System and Network Views for Intrusion Detection

paper_content:

We present several ways to correlate security events from two applications that visualize the same underlying data with two distinct views: system and network. Correlation of security events provide Security Engineers a better understanding of what is happening for enhanced security situational awareness. Visualization leverages human cognitive abilities and promotes quick mental connections between events that otherwise may be obscured in the volume of IDS alert messages.
---
paper_title: Network specific false alarm reduction in intrusion detection system

paper_content:

Intrusion Detection Systems (IDSs) are used to find the security violations in computer networks. Usually IDSs produce a vast number of alarms that include a large percentage of false alarms. One of the main reason for such false alarm generation is that, in most cases IDSs are run with default set of signatures. In this paper, a scheme for network specific false alarm reduction in IDS is proposed. A threat profile of the network is created and IDS generated alarms are correlated using neural network. Experiments conducted in a test bed have successfully filtered out most of the false alarms for a range of attacks yet maintaining the Detection Rate. Copyright © 2010 John Wiley & Sons, Ltd.
---
paper_title: NVisionIP: netflow visualizations of system state for security situational awareness

paper_content:

The number of attacks against large computer systems is currently growing at a rapid pace. Despite the best efforts of security analysts, large organizations are having trouble keeping on top of the current state of their networks. In this paper, we describe a tool called NVisionIP that is designed to increase the security analyst's situational awareness. As humans are inherently visual beings, NVisionIP uses a graphical representation of a class-B network to allow analysts to quickly visualize the current state of their network. We present an overview of NVisionIP along with a discussion of various types of security-related scenarios that it can be used to detect.
---
paper_title: Using Contextual Information for IDS Alarm Classification (Extended Abstract)

paper_content:

Signature-based intrusion detection systems are known to generate many noncritical alarms (alarms not related to a successful attack). Adding contextual information to IDSes is a promising avenue to identify noncritical alarms. Several approaches using contextual information have been suggested. However, it is not clear what are the benefits of using a specific approach. This paper establishes the effectiveness of using target configuration (i.e. operating system and applications) as contextual information for identifying noncritical alarms. Moreover, it demonstrates that current tools for OS discovery are not adequate for IDS context gathering.
---
paper_title: An Intrusion-Detection Model

paper_content:

A model of a real-time intrusion-detection expert system capable of detecting break-ins, penetrations, and other forms of computer abuse is described. The model is based on the hypothesis that security violations can be detected by monitoring a system's audit records for abnormal patterns of system usage. The model includes profiles for representing the behavior of subjects with respect to objects in terms of metrics and statistical models, and rules for acquiring knowledge about this behavior from audit records and for detecting anomalous behavior. The model is independent of any particular system, application environment, system vulnerability, or type of intrusion, thereby providing a framework for a general-purpose intrusion-detection expert system.
---
paper_title: MapReduce: simplified data processing on large clusters

paper_content:

MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day.
---

Title: Review: False alarm minimization techniques in signature-based intrusion detection systems: A survey

Section 1: Introduction
Description 1: Introduce the need for false alarm minimization in signature-based intrusion detection systems and the layout of the paper.

Section 2: Prior Work
Description 2: Discuss previous surveys and related works on false alarm minimization techniques in IDS.

Section 3: Overview
Description 3: Present an overview of various approaches proposed for false alarm minimization in signature-based IDS.

Section 4: Signature Enhancement
Description 4: Describe techniques to enhance IDS signatures with context information to reduce false alarms.

Section 5: Stateful Signatures
Description 5: Discuss the use of stateful signatures which analyze the state of the network to minimize false alarms.

Section 6: Vulnerability Signatures
Description 6: Explain the concept of using vulnerability signatures based on application semantics to improve detection and reduce false alarms.

Section 7: Alarm Mining
Description 7: Discuss various data mining techniques like clustering, classification, and frequent pattern mining used to minimize false alarms.

Section 8: Alarm Correlation
Description 8: Elaborate on different alarm correlation techniques to aggregate alarms and reconstruct attack scenarios for false alarm minimization.

Section 9: Alarm Verification
Description 9: Explain methods to verify the success of an attack corresponding to an alarm as a means to reduce false alarms.

Section 10: Flow Analysis
Description 10: Describe flow analysis used to study alarm patterns under normal and abnormal scenarios.

Section 11: Alarm Prioritization
Description 11: Discuss techniques that rate and prioritize alarms based on post-assessment or evaluation to reduce false alarms.

Section 12: Hybrid Approach
Description 12: Introduce hybrid approaches which combine multiple techniques to improve the detection accuracy and minimize false alarms.

Section 13: Commercial Tools
Description 13: Review commercial SIEM tools, their features, and performance in handling and minimizing false alarms.

Section 14: Research Questions
Description 14: Enlist research questions that need to be addressed for future improvements in false alarm minimization techniques.

Section 15: Conclusion
Description 15: Summarize the survey, discuss the benefits of various techniques, and propose future directions for research in this domain.