Intrusion detection and Big Heterogeneous Data: a Survey
---
paper_title: A survey of coordinated attacks and collaborative intrusion detection

paper_content:

Coordinated attacks, such as large-scale stealthy scans, worm outbreaks and distributed denial-of-service (DDoS) attacks, occur in multiple networks simultaneously. Such attacks are extremely difficult to detect using isolated intrusion detection systems (IDSs) that monitor only a limited portion of the Internet. In this paper, we summarize the current research directions in detecting such attacks using collaborative intrusion detection systems (CIDSs). In particular, we highlight two main challenges in CIDS research: CIDS architectures and alert correlation algorithms. We review the current CIDS approaches in terms of these two challenges. We conclude by highlighting opportunities for an integrated solution to large-scale collaborative intrusion detection.
---
paper_title: The role of big data in improving power system operation and protection

paper_content:

This paper focuses on the use of extremely large data sets in power system operation, control, and protection, which are difficult to process with traditional database tools and often termed big data. We will discuss three aspects of using such data sets: feature extraction, systematic integration for power system applications, and examples of typical applications in the utility industry. The following analytics tasks based on big data methodology are elaborated upon: corrective, predictive, distributed and adaptive. The paper also outlines several research topics related to asset management, operation planning, realtime monitoring and fault detection/protection that present new opportunities but require further investigation.
---
paper_title: Secure Outsourcing of Network Flow Data Analysis

paper_content:

In this paper, we identify a new and challenging application for the growing field of research on data anonymization and secure outsourcing of storage and computations to the cloud. Network flow data analysis is of high importance for network monitoring and management. Network monitoring applications reveal new challenges not yet addressed in the secure outsourcing literature. The secure and verifiable outsourcing of computation on anonymized network flow records provides a practical tool for network operators in order to harness the cloud benefits, which untapped until now because of privacy concerns. We present representative use-cases and problems, and identify sample related work that can be utilized for developing an effective solution.
---
paper_title: A survey of intrusion detection techniques in Cloud

paper_content:

In this paper, we survey different intrusions affecting availability, confidentiality and integrity of Cloud resources and services. Proposals incorporating Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) in Cloud are examined. We recommend IDS/IPS positioning in Cloud environment to achieve desired security in the next generation networks.
---
paper_title: An Intrusion-Detection Model

paper_content:

A model of a real-time intrusion-detection expert system capable of detecting break-ins, penetrations, and other forms of computer abuse is described. The model is based on the hypothesis that security violations can be detected by monitoring a system's audit records for abnormal patterns of system usage. The model includes profiles for representing the behavior of subjects with respect to objects in terms of metrics and statistical models, and rules for acquiring knowledge about this behavior from audit records and for detecting anomalous behavior. The model is independent of any particular system, application environment, system vulnerability, or type of intrusion, thereby providing a framework for a general-purpose intrusion-detection expert system.
---
paper_title: Snort - Lightweight Intrusion Detection for Networks

paper_content:

Network intrusion detection systems (NIDS) are an important part of any network security architecture. They provide a layer of defense which monitors network traffic for predefined suspicious activity or patterns, and alert system administrators when potential hostile traffic is detected. Commercial NIDS have many differences, but Information Systems departments must face the commonalities that they share such as significant system footprint, complex deployment and high monetary cost. Snort was designed to address these issues.
---
paper_title: Mining intrusion detection alarms for actionable knowledge

paper_content:

In response to attacks against enterprise networks, administrators increasingly deploy intrusion detection systems. These systems monitor hosts, networks, and other resources for signs of security violations. The use of intrusion detection has given rise to another difficult problem, namely the handling of a generally large number of alarms. In this paper, we mine historical alarms to learn how future alarms can be handled more efficiently. First, we investigate episode rules with respect to their suitability in this approach. We report the difficulties encountered and the unexpected insights gained. In addition, we introduce a new conceptual clustering technique, and use it in extensive experiments with real-world data to show that intrusion detection alarms can be handled efficiently by using previously mined knowledge.
---
paper_title: Emerging trends around big data analytics and security: panel

paper_content:

This panel will discuss the interplay between key emerging security trends centered around big data analytics and security. With the explosion of big data and advent of cloud computing, data analytics has not only become prevalent but also a critical business need. Internet applications today consume vast amounts of data collected from heterogeneous big data repositories and provide meaningful insights from it. These include applications for business forecasting, investment and finance, healthcare and well-being, science and hi-tech, to name a few. Security and operational intelligence is one of the critical areas where big data analytics is expected to play a crucial role. Security analytics in a big data environment presents a unique set of challenges, not properly addressed by the existing security incident and event monitoring (or SIEM) systems that typically work with a limited set of traditional data sources (firewall, IDS, etc.) in an enterprise network. A big data environment presents both a great opportunity and a challenge due to the explosion and heterogeneity of the potential data sources that extend the boundary of analytics to social networks, real time streams and other forms of highly contextual data that is characterized by high volume and speed. In addition to meeting infrastructure challenges, there remain additional unaddressed issues, including but not limited to development of self-evolving threat ontologies, integrated network and application layer analytics, and detection of "low and slow" attacks. At the same time, security analytics requires a high degree of data assurance, where assurance implies that the data be trustworthy as well as managed in a privacy preserving manner. Our panelists represent individuals from industry, academia, and government who are at the forefront of big data security analytics. They will provide insights into these unique challenges, survey the emerging trends, and lay out a vision for future.
---
paper_title: Relevance feature selection with data cleaning for intrusion detection system

paper_content:

Labeled datasets play a major role in the process of validating and evaluating machine learning techniques in intrusion detection systems. In order to obtain good accuracy in the evaluation, very large datasets should be considered. Intrusion traffic and normal traffic are in general dependent on a large number of network characteristics called features. However not all of these features contribute to the traffic characteristics. Therefore, eliminating the non-contributing features from the datasets, to facilitate speed and accuracy to the evaluation of machine learning techniques, becomes an important requirement. In this paper we suggest an approach which analyzes the intrusion datasets, evaluates the features for its relevance to a specific attack, determines the level of contribution of feature, and eliminates it from the dataset automatically. We adopt the Rough Set Theory (RST) based approach and select relevance features using multidimensional scatter-plot automatically. A pair-wise feature selection process is adopted to simplify. In our previous research we used KDD'99 dataset and validated the RST based approach. There are lots of redundant data entries in KDD'99 and thus the machine learning techniques are biased towards most occurring events. This property leads the algorithms to ignore less frequent events which can be more harmful than most occurring events. False positives are another important drawback in KDD'99 dataset. In this paper, we adopt NSL-KDD dataset (an improved version of KDD'99 dataset) and validate the automated RST based approach. The approach presented in this paper leads to a selection of most relevance features and we expect that the intrusion detection research using KDD'99-based datasets will benefit from the good understanding of network features and their influences to attacks.
---
paper_title: Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory

paper_content:

In 1998 and again in 1999, the Lincoln Laboratory of MIT conducted a comparative evaluation of intrusion detection systems (IDSs) developed under DARPA funding. While this evaluation represents a significant and monumental undertaking, there are a number of issues associated with its design and execution that remain unsettled. Some methodologies used in the evaluation are questionable and may have biased its results. One problem is that the evaluators have published relatively little concerning some of the more critical aspects of their work, such as validation of their test data. The appropriateness of the evaluation techniques used needs further investigation. The purpose of this article is to attempt to identify the shortcomings of the Lincoln Lab effort in the hope that future efforts of this kind will be placed on a sounder footing. Some of the problems that the article points out might well be resolved if the evaluators were to publish a detailed description of their procedures and the rationale that led to their adoption, but other problems would clearly remain./par>
---
paper_title: Playing Devil's Advocate: Inferring Sensitive Information from Anonymized Network Traces

paper_content:

Encouraging the release of network data is central to promoting sound network research practices, though the publication of this data can leak sensitive information about the publishing organization. To address this dilemma, several techniques have been suggested for anonymizing network data by obfuscating sensitive fields. In this paper, we present new techniques for inferring network topology and deanonymizing servers present in anonymized network data, using only the data itself and public information. Via analyses on three different network datasets, we quantify the effectiveness of our techniques, showing that they can uncover significant amounts of sensitive information. We also discuss prospects for preventing these deanonymization attacks.
---
paper_title: Outside the Closed World: On Using Machine Learning for Network Intrusion Detection

paper_content:

In network intrusion detection research, one popular strategy for finding attacks is monitoring a network's activity for anomalies: deviations from profiles of normality previously learned from benign traffic, typically identified using tools borrowed from the machine learning community. However, despite extensive academic research one finds a striking gap in terms of actual deployments of such systems: compared with other intrusion detection approaches, machine learning is rarely employed in operational "real world" settings. We examine the differences between the network intrusion detection problem and other areas where machine learning regularly finds much more success. Our main claim is that the task of finding attacks is fundamentally different from these other applications, making it significantly harder for the intrusion detection community to employ machine learning effectively. We support this claim by identifying challenges particular to network intrusion detection, and provide a set of guidelines meant to strengthen future research on anomaly detection.
---
paper_title: Data Mining in Intrusion Detection: A Comparative Study of Methods, Types and Data Sets

paper_content:

In the era of information and communication technology, Security is an important issue. A lot of effort and finance are being invested in this sector. Intrusion detection is one of the most prominent fields in this area. Data mining in network intrusion detection can automate the network intrusion detection field with a greater efficiency. This paper presents a literature survey on intrusion detection system. The research papers taken in this literature survey are published from 2000 to 2012. We can see that almost 67 % of the research papers are focused on anomaly detection, 23 % on both anomaly and misuse detection and 10 % on misuse detection. In this literature survey statistics shows that 42 % KDD cup dataset, 20 % DARPA dataset and 38 % other datasets are used by the different researchers for testing the effectiveness of their proposed method for misuse detection, anomaly detection or both.
---
paper_title: Toward developing a systematic approach to generate benchmark datasets for intrusion detection

paper_content:

In network intrusion detection, anomaly-based approaches in particular suffer from accurate evaluation, comparison, and deployment which originates from the scarcity of adequate datasets. Many such datasets are internal and cannot be shared due to privacy issues, others are heavily anonymized and do not reflect current trends, or they lack certain statistical characteristics. These deficiencies are primarily the reasons why a perfect dataset is yet to exist. Thus, researchers must resort to datasets that are often suboptimal. As network behaviors and patterns change and intrusions evolve, it has very much become necessary to move away from static and one-time datasets toward more dynamically generated datasets which not only reflect the traffic compositions and intrusions of that time, but are also modifiable, extensible, and reproducible. In this paper, a systematic approach to generate the required datasets is introduced to address this need. The underlying notion is based on the concept of profiles which contain detailed descriptions of intrusions and abstract distribution models for applications, protocols, or lower level network entities. Real traces are analyzed to create profiles for agents that generate real traffic for HTTP, SMTP, SSH, IMAP, POP3, and FTP. In this regard, a set of guidelines is established to outline valid datasets, which set the basis for generating profiles. These guidelines are vital for the effectiveness of the dataset in terms of realism, evaluation capabilities, total capture, completeness, and malicious activity. The profiles are then employed in an experiment to generate the desirable dataset in a testbed environment. Various multi-stage attacks scenarios were subsequently carried out to supply the anomalous portion of the dataset. The intent for this dataset is to assist various researchers in acquiring datasets of this kind for testing, evaluation, and comparison purposes, through sharing the generated datasets and profiles.
---
paper_title: MAWILab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking

paper_content:

Evaluating anomaly detectors is a crucial task in traffic monitoring made particularly difficult due to the lack of ground truth. The goal of the present article is to assist researchers in the evaluation of detectors by providing them with labeled anomaly traffic traces. We aim at automatically finding anomalies in the MAWI archive using a new methodology that combines different and independent detectors. A key challenge is to compare the alarms raised by these detectors, though they operate at different traffic granularities. The main contribution is to propose a reliable graph-based methodology that combines any anomaly detector outputs. We evaluated four unsupervised combination strategies; the best is the one that is based on dimensionality reduction. The synergy between anomaly detectors permits to detect twice as many anomalies as the most accurate detector, and to reject numerous false positive alarms reported by the detectors. Significant anomalous traffic features are extracted from reported alarms, hence the labels assigned to the MAWI archive are concise. The results on the MAWI traffic are publicly available and updated daily. Also, this approach permits to include the results of upcoming anomaly detectors so as to improve over time the quality and variety of labels.
---
paper_title: Statistical analysis of honeypot data and building of Kyoto 2006+ dataset for NIDS evaluation

paper_content:

With the rapid evolution and proliferation of botnets, large-scale cyber attacks such as DDoS, spam emails are also becoming more and more dangerous and serious cyber threats. Because of this, network based security technologies such as Network based Intrusion Detection Systems (NIDSs), Intrusion Prevention Systems (IPSs), firewalls have received remarkable attention to defend our crucial computer systems, networks and sensitive information from attackers on the Internet. In particular, there has been much effort towards high-performance NIDSs based on data mining and machine learning techniques. However, there is a fatal problem in that the existing evaluation dataset, called KDD Cup 99' dataset, cannot reflect current network situations and the latest attack trends. This is because it was generated by simulation over a virtual network more than 10 years ago. To the best of our knowledge, there is no alternative evaluation dataset. In this paper, we present a new evaluation dataset, called Kyoto 2006+, built on the 3 years of real traffic data (Nov. 2006 ~ Aug. 2009) which are obtained from diverse types of honeypots. Kyoto 2006+ dataset will greatly contribute to IDS researchers in obtaining more practical, useful and accurate evaluation results. Furthermore, we provide detailed analysis results of honeypot data and share our experiences so that security researchers are able to get insights into the trends of latest cyber attacks and the Internet situations.
---
paper_title: Survey and taxonomy of feature selection algorithms in intrusion detection system

paper_content:

The Intrusion detection system deals with huge amount of data which contains irrelevant and redundant features causing slow training and testing process, higher resource consumption as well as poor detection rate. Feature selection, therefore, is an important issue in intrusion detection. In this paper we introduce concepts and algorithms of feature selection, survey existing feature selection algorithms in intrusion detection systems, group and compare different algorithms in three broad categories: filter, wrapper, and hybrid. We conclude the survey by identifying trends and challenges of feature selection research and development in intrusion detection system.
---
paper_title: Genetic-fuzzy rule mining approach and evaluation of feature selection techniques for anomaly intrusion detection

paper_content:

Classification of intrusion attacks and normal network traffic is a challenging and critical problem in pattern recognition and network security. In this paper, we present a novel intrusion detection approach to extract both accurate and interpretable fuzzy IF-THEN rules from network traffic data for classification. The proposed fuzzy rule-based system is evolved from an agent-based evolutionary framework and multi-objective optimization. In addition, the proposed system can also act as a genetic feature selection wrapper to search for an optimal feature subset for dimensionality reduction. To evaluate the classification and feature selection performance of our approach, it is compared with some well-known classifiers as well as feature selection filters and wrappers. The extensive experimental results on the KDD-Cup99 intrusion detection benchmark data set demonstrate that the proposed approach produces interpretable fuzzy systems, and outperforms other classifiers and wrappers by providing the highest detection accuracy for intrusion attacks and low false alarm rate for normal network traffic with minimized number of features.
---
paper_title: Feature deduction and ensemble design of intrusion detection systems

paper_content:

Current intrusion detection systems (IDS) examine all data features to detect intrusion or misuse patterns. Some of the features may be redundant or contribute little (if anything) to the detection process. The purpose of this study is to identify important input features in building an IDS that is computationally efficient and effective. We investigated the performance of two feature selection algorithms involving Bayesian networks (BN) and Classification and Regression Trees (CART) and an ensemble of BN and CART. Empirical results indicate that significant input feature selection is important to design an IDS that is lightweight, efficient and effective for real world detection systems. Finally, we propose an hybrid architecture for combining different feature selection algorithms for real world intrusion detection.
---
paper_title: Big data classification: problems and challenges in network intrusion prediction with machine learning

paper_content:

This paper focuses on the specific problem of Big Data classification of network intrusion traffic. It discusses the system challenges presented by the Big Data problems associated with network intrusion prediction. The prediction of a possible intrusion attack in a network requires continuous collection of traffic data and learning of their characteristics on the fly. The continuous collection of traffic data by the network leads to Big Data problems that are caused by the volume, variety and velocity properties of Big Data. The learning of the network characteristics require machine learning techniques that capture global knowledge of the traffic patterns. The Big Data properties will lead to significant system challenges to implement machine learning frameworks. This paper discusses the problems and challenges in handling Big Data classification using geometric representation-learning techniques and the modern Big Data networking technologies. In particular this paper discusses the issues related to combining supervised learning techniques, representation-learning techniques, machine lifelong learning techniques and Big Data technologies (e.g. Hadoop, Hive and Cloud) for solving network traffic classification problems.
---
paper_title: A Real-time Intrusion Detection System by Integrating Hadoop and Naive Bayes Classification

paper_content:

There is a meteoric rise in intrusions that penetrate highly secure networks. Furthermore, a successful intrusion serves as a motivation factor for aspiring black-hat hackers to launch their own intrusion campaigns. Security experts are now facing the complications of growing network data generated by their networks. Almost all businesses have an online platform through which multitudes of companies have reaped enormous profits. Hence, such networks that churn out terabytes/gigabytes of data (big data) remain as an indispensable asset for companies. To process the huge network traffic data we require powerful machines that are often too expensive. In this paper, we present our intrusion detection system that employs a Naive Bayes algorithm that runs in a distributed manner. The classifier uses the Apache Hadoop and HStreaming APIs to detect intrusions in real time. The network traffic data generated by non-commercial packet sniffers serve as input to our classifier. Our aim is to develop a distributed, scalable, fault tolerant and reliable system. The system is designed in such way that it can run effortlessly on a cluster comprised of obsolete hardware. The results denote that our system's performance is acceptable in real case scenarios.
---
paper_title: Security problems and challenges in a machine learning-based hybrid big data processing network systems

paper_content:

The data source that produces data continuously in high volume and high velocity with large varieties of data types creates Big Data, and causes problems and challenges to Machine Learning (ML) techniques that help extract, analyze and visualize important information. To overcome these problems and challenges, we propose to make use of the hybrid networking model that consists of multiple components such as Hadoop distributed file system (HDFS), cloud storage system, security module and ML unit. Processing of Big Data in this networking environment with ML technique requires user interaction and additional storage hence some artificial delay between the arrivals of data domains through external storage can help HDFSto process the Big Data efficiently. To address this problem we suggest using public cloud for data storage which will induce meaningful time delay to the data while making use of its storage capability. However, the use of public cloud will lead to security vulnerability to the data transmission and storage. Therefore, we need some form of security algorithm that provides a flexible key-based encryption technique that can provide tradeoffs between time-delay, security strength and storage risks. In this paper we propose a model for using public cloud provider trust levels to select encryption types for data storage for use within a Big Data analytics network topology.
---
paper_title: Anomaly Teletraffic Intrusion Detection Systems on Hadoop-Based Platforms: A Survey of Some Problems and Solutions

paper_content:

Telecommunication networks are getting more important in our social lives because many people want to share their information and ideas. Thanks to the rapid development of the Internet and ubiquitous technologies including mobile devices such as smart phones, mobile phones and tablet PCs, the quality of our lives has been greatly influenced and rapidly changed in recent years. Internet users have exponentially increased as well. Meanwhile, the explosive growth of teletraffic called big data for user services threatens the current networks, and we face menaces from various kinds of intrusive incidents through the Internet. A variety of network attacks on network resources have continuously caused serious damage. Thus, active and advanced technologies for early detecting of anomaly teletraffic on Hadoop-based platforms are required. In this paper, a survey of some problems and technical solutions for anomaly teletraffic intrusion detection systems based on the open-source software platform Hadoop has been investigated and proposed.
---
paper_title: Distributed Processing of Snort Alert Log using Hadoop

paper_content:

Snort is a famous tool for Intrusion Detection System (IDS), which is used to gather and analyse network packet in order to decide attacks through network. Until now, although processing a number of warning messages in real time, Snort is executed mainly in single computer systems. Unfortunately, current amount of network messages exceeds processing capacity of single computer systems. In order to embrace the huge amount of network messages, we have constructed a distributed IDS using Hadoop, HDFS, and 8 working nodes. Experimental results show that our distributed IDS has 426% performance compared to a single computer system. Keyword-Intrusion Detection System, Snort, Distributed Framework, Hadoop, HDFS
---
paper_title: Toward scalable internet traffic measurement and analysis with Hadoop

paper_content:

Internet traffic measurement and analysis has long been used to characterize network usage and user behaviors, but faces the problem of scalability under the explosive growth of Internet traffic and high-speed access. Scalable Internet traffic measurement and analysis is difficult because a large data set requires matching computing and storage resources. Hadoop, an open-source computing platform of MapReduce and a distributed file system, has become a popular infrastructure for massive data analytics because it facilitates scalable data processing and storage services on a distributed computing system consisting of commodity hardware. In this paper, we present a Hadoop-based traffic monitoring system that performs IP, TCP, HTTP, and NetFlow analysis of multi-terabytes of Internet traffic in a scalable manner. From experiments with a 200-node testbed, we achieved 14 Gbps throughput for 5 TB files with IP and HTTP-layer analysis MapReduce jobs. We also explain the performance issues related with traffic analysis MapReduce jobs.
---
paper_title: A framework for network security situation awareness based on knowledge discovery

paper_content:

Network security situation awareness provides the unique high level security view based upon the security alert events. But the complexities and diversities of security alert data on modern networks make such analysis extremely difficult. In this paper, we analyze the existing problems of network security situation awareness system and propose a framework for network security situation awareness based on knowledge discovery. The framework consists of the modeling of network security situation and the generation of network security situation. The purpose of modeling is to construct the formal model of network security situation measurement based upon the D-S evidence theory, and support the general process of fusing and analyzing security alert events collected from security situation sensors. The generation of network security situation is to extract the frequent patterns and sequential patterns from the dataset of network security situation based upon knowledge discovery method and transform these patterns to the correlation rules of network security situation, and finally to automatically generate the network security situation graph. Application of the integrated Network Security Situation Awareness system (Net-SSA) shows that the proposed framework supports for the accurate modeling and effective generation of network security situation.
---
paper_title: An introduction to multisensor data fusion

paper_content:

Multisensor data fusion is an emerging technology applied to Department of Defense (DoD) areas such as automated target recognition, battlefield surveillance, and guidance and control of autonomous vehicles, and to non-DoD applications such as monitoring of complex machinery, medical diagnosis, and smart buildings. Techniques for multisensor data fusion are drawn from a wide range of areas including artificial intelligence, pattern recognition, statistical estimation and other areas. This paper provides a tutorial on data fusion, introducing data fusion applications, process models, and identification of applicable techniques. Comments are made on the state-of-the-art in data fusion.
---
paper_title: Data Fusion: Concepts and Ideas

paper_content:

Introduction.- Sensors.- Architecture.- Common Representational Format.- Spatial Alignment.- Temporal Alignment.- Semantic Alignment.- Radiometric Normalization.- Bayesian Inference.- Parameter Estimation.- Robust Statistics.- Sequential Bayesian Inference.- Bayesian Decision Theory.- Ensemble Learning.- Sensor Management.
---
paper_title: Collaborative Intrusion Detection Framework: Characteristics, Adversarial Opportunities and Countermeasures

paper_content:

Complex Internet attacks may come from multiple sources, and target multiple networks and technologies. Nevertheless, Collaborative Intrusion Detection Systems (CIDS) emerges as a promising solution by using information from multiple sources to gain a better understanding of objective and impact of complex Internet attacks. CIDS also help to cope with classical problems of Intrusion Detection Systems (IDS) such as zero-day attacks, high false alarm rates and architectural challenges, e. g., centralized designs exposing the Single-Point-of-Failure. Improved complexity on the other hand gives raise to new exploitation opportunities for adversaries. ::: ::: The contribution of this paper is twofold. We first investigate related research on CIDS to identify the common building blocks and to understand vulnerabilities of the Collaborative Intrusion Detection Framework (CIDF). Second, we focus on the problem of anonymity preservation in a decentralized intrusion detection related message exchange scheme. We use techniques from design theory to provide multi-path peer-to-peer communication scheme where the adversary can not perform better than guessing randomly the originator of an alert message.
---
paper_title: Design and implementation of a DIDS

paper_content:

With the fast development of computer network, the problem of network security becomes more and more serious. The single firewall can not solve the problem entirely and an intrusion detection system is helpful. This paper presents the design and development of a distributed intrusion detection system to protect the network security. Experiments shows that it works well.
---
paper_title: Data collection for information security system

paper_content:

The security of information systems is generally related to the deployed tools to protect the network infrastructure from attacks. These latter are based on detection components that analyze the information transmitted through the network. The existing systems do not provide a full protection and present a high rate of false positives that could result from a lack in the analysis of the collected information (events). Thus, useful information should be determined to perform accurate detection and hence perform good protection. In this paper, a new approach to collect useful data from the network is proposed. It is based on new components called: observers. The system architecture is presented and the different components are described in details. Furthermore, the functioning of the different parts of the system is discussed.
---
paper_title: Integrated Security Incident Management -- Concepts and Real-World Experiences

paper_content:

We present a holistic, process-oriented approach to ISO/IEC 27001 compliant security incident management that integrates multiple state-of-the-art security tools and has been applied to a real-world scenario very successfully for one year so far. The computer security incident response team, CSIRT, is enabled to correlate IT security related events across multiple communication channels and thus to classify any incidents consistently. Depending on an incident's classification, manual intervention or even fully automated reaction steps can be triggered, this starts with simple email notifications of system and network administrators, and scales up to quarantining compromised systems and sub networks automatically. A formally specified security incident response (SIR) process serves as the basis that clearly defines responsibilities, workflows, and interfaces. It has been designed to enable quick reactions to IT security events in a very resource-conserving manner.
---
paper_title: A Global Security Architecture for Intrusion Detection on Computer Networks

paper_content:

Detecting all kinds of intrusions efficiently requires a global view of the monitored network. Built to increase the security of computer networks, traditional IDS are unfortunately unable to give a global view of the security of a network. To overcome this situation, we are developing a distributed SOC (security operation center) which is able to detect attacks occurring simultaneously on several sites in a network and to give a global view of the security of that network. In this article, we present the global architecture of our system, called DSOC as well as several methods used to test its accuracy and performance.
---
paper_title: Decentralized multi-dimensional alert correlation for collaborative intrusion detection

paper_content:

The growth in coordinated network attacks such as scans, worms and distributed denial-of-service (DDoS) attacks is a profound threat to the security of the Internet. Collaborative intrusion detection systems (CIDSs) have the potential to detect these attacks, by enabling all the participating intrusion detection systems (IDSs) to share suspicious intelligence with each other to form a global view of the current security threats. Current correlation algorithms in CIDSs are either too simple to capture the important characteristics of attacks, or too computationally expensive to detect attacks in a timely manner. We propose a decentralized, multi-dimensional alert correlation algorithm for CIDSs to address these challenges. A multi-dimensional alert clustering algorithm is used to extract the significant intrusion patterns from raw intrusion alerts. A two-stage correlation algorithm is used, which first clusters alerts locally at each IDS, before reporting significant alert patterns to a global correlation stage. We introduce a probabilistic approach to decide when a pattern at the local stage is sufficiently significant to warrant correlation at the global stage. We then implement the proposed two-stage correlation algorithm in a fully distributed CIDS. Our experiments on a large real-world intrusion data set show that our approach can achieve a significant reduction in the number of alert messages generated by the local correlation stage with negligible false negatives compared to a centralized scheme. The proposed probabilistic threshold approach gains a significant improvement in detection accuracy in a stealthy attack scenario, compared to a naive scheme that uses the same threshold at the local and global stages. A large scale experiment on PlanetLab shows that our decentralized architecture is significantly more efficient than a centralized approach in terms of the time required to correlate alerts.
---
paper_title: Self-Organized Mechanism for Distributed Setup of Multiple Heterogeneous Intrusion Detection Systems

paper_content:

We propose a distributed and self-organized framework for collaboration of multiple heterogeneous IDS sensors. The framework is based on a game-theoretical approach that optimizes behavior of each IDS sensor with respect to other sensors in highly dynamic environments. We formalize the proposed collaborative architecture as a game between defenders and attackers and transform the hard problem of heterogeneous collaboration into an easier problem of finding two functions that are used in the game-theoretical model to specialize the detection mechanisms on a specific type of malicious activity. The collaboration of such more specialized IDS nodes covers much wider range of attack classes, allowing the collaborating system to maximize the overall network security awareness. We have evaluated the proposed concept on real networks, where we have shown considerable improvements in the detection capabilities of intrusion detection devices thanks to the proposed collaboration model.
---
paper_title: Managing Security: The Security Content Automation Protocol

paper_content:

Managing information systems security is an expensive and challenging task. Many different and complex software components- including firmware, operating systems, and applications-must be configured securely, patched when needed, and continuously monitored for security. Most organizations have an extensive set of security requirements. For commercial firms, such requirements are established through complex interactions of business goals, government regulations, and insurance requirements; for government organizations, security requirements are mandated. Meeting these requirements has been time consuming and error prone, because organizations have lacked standardized, automated ways of performing the tasks and reporting on results. To overcome these deficiencies and reduce security administration costs, the National Institute of Standards and Technology developed the security content automation protocol using community supported security resources. SCAP (pronounced "ess-cap") is a suite of specifications that standardizes the format and nomenclature by which security software products communicate information about software identification, software flaws, and security configurations.
---
paper_title: Integrated Security Incident Management -- Concepts and Real-World Experiences

paper_content:

We present a holistic, process-oriented approach to ISO/IEC 27001 compliant security incident management that integrates multiple state-of-the-art security tools and has been applied to a real-world scenario very successfully for one year so far. The computer security incident response team, CSIRT, is enabled to correlate IT security related events across multiple communication channels and thus to classify any incidents consistently. Depending on an incident's classification, manual intervention or even fully automated reaction steps can be triggered, this starts with simple email notifications of system and network administrators, and scales up to quarantining compromised systems and sub networks automatically. A formally specified security incident response (SIR) process serves as the basis that clearly defines responsibilities, workflows, and interfaces. It has been designed to enable quick reactions to IT security events in a very resource-conserving manner.
---
paper_title: Improving the Automation of Security Information Management: A Collaborative Approach

paper_content:

Many preventive security measures purport to protect networks from cyber intrusions. These adopted measures can generate a large amount of information that should be stored and analyzed to enable responses to detected attacks. Security information and event managers (SIEMs) are indispensable for collecting all of a system's security-related information in a central repository. This can then provide trend analysis and lead analysts to adopt appropriate actions. A collaborative work approach lets SIEMs of different trusted domains share alarms and their countermeasures. By sharing alarms and adopted measures in domains with similar profiles, the authors hope to enhance a global view of the security and facilitate decision making for security-domain administrators.
---
paper_title: The Ontological Approach for SIEM Data Repository Implementation

paper_content:

The technology of Security Information and Event Management (SIEM) becomes one of the most important research applications in the area of computer network security, including distributed networks of internet enabled objects (as in the Internet of Things). The overall functionality of SIEM systems depends largely on the quality of solutions implemented at the data storage level, which is purposed for the representation of heterogeneous security events, their storage in the data repository and the extraction of relevant data for the analytical modules of SIEM systems. An ontological approach at present becomes more applicable for realizing these tasks in various spheres of information security. The paper discusses the possibilities of applying the ontological approach for implementation of the data repository of SIEM systems for distributed networks of Internet enabled objects. Based on the analysis of existing SIEM systems and standards, the choice of ontological approach is done, an example of the ontological data model of vulnerabilities is presented, a hybrid architecture of the ontological repository is proposed and the issues of developing and testing the repository for attack modelling and secure evaluation tasks are discussed.
---
paper_title: Cross-Boundary Enterprise Security Monitoring

paper_content:

The cyber threat landscape is evolving at a much faster rate than corporate cyber defenses are. Malicious actors have become more audacious in their exploit attempts during the past two years as evidenced by the growing number of publicly acknowledged breaches in the commercial and industrial sectors. Motivations range the gamut from “hacktivism” to corporate espionage. This paper will discuss one possible approach to defending against malicious actors at an enterprise level. The proposed approach is based on cross-boundary information security management and the fusion of enterprise-wide security data.
---
paper_title: An investigation and survey of response options for Intrusion Response Systems (IRSs)

paper_content:

The rise of attacks and incidents need additional and distinct methods of response. This paper starts a discussion by differentiating the type of operation mode such as Intrusion Detection Systems (IDSs), Intrusion Prevention Systems (IPSs) and Intrusion Response Systems (IRSs). Using characteristics of response and attack time frame, a response model is proposed to distinguish between active and passive response options. The characteristics of response include level of operations, speed and time of response, ability to learn and ability to cooperate with other devices. This paper uses the attack time frame as a response model to show the relationship between active and passive response. Furthermore, the Response Model for Intrusion Response Systems shows some other different approaches and stages of active response. Finally, in order to investigate the most common response used by security practitioner and to justify the response model, studies involving 34 samples products from both commercial and non-commercial are analysed. As a result, this paper shows a clear distinction between the options of responses.
---
paper_title: Common Framework for Attack Modeling and Security Evaluation in SIEM Systems

paper_content:

The paper suggests a framework for attack modeling and security evaluation in Security Information and Event Management (SIEM) systems. It is supposed that the common approach to attack modeling and security evaluation is based on modeling of a malefactor's behavior, generating a common attack graph, calculating different security metrics and providing risk analysis procedures. Key elements of suggested architectural solutions for attack modeling and security evaluation are using a comprehensive security repository, effective attack graph (tree) generation techniques, taking into account known and new attacks based on zero-day vulnerabilities, stochastic analytical modeling, and interactive decision support to choose preferred security solutions. The architecture of the Attack Modeling and Security Evaluation Component (AMSEC) is proposed, its interaction with other SIEM components is described. We present the prototype of the component and the results of experiments carried out.
---
paper_title: A trustworthy and resilient event broker for monitoring cloud infrastructures

paper_content:

We propose a fault- and intrusion-tolerant framework for datacenter and cloud infrastructure monitoring. In contrast to existing approaches, our framework is able to deal with Byzantine faults. It is based on a replicated event broker, lying in the core of the monitoring infrastructure, supporting the dissemination of all monitoring events. We describe the architecture and the behavior of the framework, explaining how we can deal with different requirements on QoS and QoP. We provide evaluation results as proof of concept for the proposed framework.
---
paper_title: Log-Based Distributed Security Event Detection Using Simple Event Correlator

paper_content:

Log event correlation is an effective means of detecting system faults and security breaches encountered in information technology environments. Centralized, database-driven log event correlation is common, but suffers from flaws such as high network bandwidth utilization, significant requirements for system resources, and difficulty in detecting certain suspicious behaviors. Distributed event correlation is often assumed to be superior, but no research effort has been made which quantitatively evaluates its advantages and disadvantages. This research presents a distributed event correlation system which performs security event detection, and evaluates it experimentally, compared with a centralized alternative. The comparison measures the value in distributed event correlation by considering network bandwidth utilization, detection capability and database query efficiency. The implementation of these advantages allows a 99% reduction of network syslog traffic in the low accountability case. In addition, the system detects every implemented malicious use case, with a low false positive rate.
---
paper_title: Intrusion Detection System for High Volume and High Velocity Packet Streams: A Clustering Approach

paper_content:

 Abstract—The success of any Intrusion Detection System lies in its ability to quickly adapt to new threats in near real time and further prevent new attacks. This implies extremely efficient machine learning algorithms in the backend, which in turn may use clustering algorithms capable of distinguishing between normal and anomalous network traffic. This work is a first step towards proposing such an IDS, which is built on clustering-based machine learning. The authors evaluate different clustering algorithms using a network packet trace and provide results, which help in evaluating these algorithms. The work-in-progress section of the paper visualizes the IDS which can be used in an environment where the traffic volumes are very high, enterprise boundaries are blurred, and the likelihood of malicious attacks is extremely high.
---
paper_title: Secure Outsourcing of Network Flow Data Analysis

paper_content:

In this paper, we identify a new and challenging application for the growing field of research on data anonymization and secure outsourcing of storage and computations to the cloud. Network flow data analysis is of high importance for network monitoring and management. Network monitoring applications reveal new challenges not yet addressed in the secure outsourcing literature. The secure and verifiable outsourcing of computation on anonymized network flow records provides a practical tool for network operators in order to harness the cloud benefits, which untapped until now because of privacy concerns. We present representative use-cases and problems, and identify sample related work that can be utilized for developing an effective solution.
---
paper_title: Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks

paper_content:

As more and more Internet-based attacks arise, organizations are responding by deploying an assortment of security products that generate situational intelligence in the form of logs. These logs often contain high volumes of interesting and useful information about activities in the network, and are among the first data sources that information security specialists consult when they suspect that an attack has taken place. However, security products often come from a patchwork of vendors, and are inconsistently installed and administered. They generate logs whose formats differ widely and that are often incomplete, mutually contradictory, and very large in volume. Hence, although this collected information is useful, it is often dirty. We present a novel system, Beehive, that attacks the problem of automatically mining and extracting knowledge from the dirty log data produced by a wide variety of security products in a large enterprise. We improve on signature-based approaches to detecting security incidents and instead identify suspicious host behaviors that Beehive reports as potential security incidents. These incidents can then be further analyzed by incident response teams to determine whether a policy violation or attack has occurred. We have evaluated Beehive on the log data collected in a large enterprise, EMC, over a period of two weeks. We compare the incidents identified by Beehive against enterprise Security Operations Center reports, antivirus software alerts, and feedback from enterprise security specialists. We show that Beehive is able to identify malicious events and policy violations which would otherwise go undetected.
---
paper_title: A big data acquisition engine based on rule engine

paper_content:

Abstract Supervisory control and data acquisition (SCADA) systems based on Ethernet progress is open and modularization. This article aims to building a general data acquisition engine which combines rule engine and finite state automaton together. The combination describes big data acquisition flow and its rules in a flexible way, and helps to verify the security and correctness of the big data acquisition flow.
---
paper_title: Intrusion Monitoring in Process Control Systems

paper_content:

To protect process control networks from cyber intrusions, preventive security measures such as perimeter defenses (for example, network firewalls and demilitarized zones) and secure versions of process control network protocols have been increasingly adopted or proposed. Although system hardening and fixing known vulnerabilities of existing systems are crucial to secure process control systems, intrusion monitoring is essential to ensure that the preventive measures are not compromised or bypassed. Our approach involves a multilayer security architecture for monitoring process control systems to achieve accurate and effective situational awareness. Also, we leverage some of the characteristics of process control systems such as the regularity of network traffic patterns to perform intrusion detection, with the potential to detect unknown attacks. To facilitate human analysts to gain a better understanding of anomalous network traffic patterns, we present a visualization tool that supports multiple usercustomizable views and animation for analyzing network packet traces.
---
paper_title: Analyzing Malware Log Data to Support Security Information and Event Management: Some Research Results

paper_content:

Enterprise information infrastructures are generally characterized by a multitude of information systems which support decision makers in fulfilling their duties. The object of information security management is the protection of these systems, whereas security information and event management (SIEM) addresses those information management tasks which focus on the short term handling of events, as well as on the long term improvement of the entire information security architectures. This is carried out based on those data which can be logged and collected within the enterprise information security infrastructure. An especially interesting type of log data is data created by anti-malware software. This paper demonstrates in the context of a project case study that data mining (DM) is a well suited approach to detect hidden patterns in malware data and thus to support SIEM.
---
paper_title: The Design of Real-Time Adaptive Forensically Sound Secure Critical Infrastructure

paper_content:

Network security design has seen significant advances in recent years. This has been demonstrated by a growing number of new encryption algorithms, more intelligent firewall and intrusion detection techniques, new developments in multifactor authentication, advances in malware protection and many more. During a similar period of time the industry has seen the need for network infrastructure which provides a greater degree of trust which has resulted in the development of forensic analysis tools which meet the requirements of law enforcement agencies. Such tools must provide for commercial intelligence and national security. This paper proposes that application of the common ground between security and forensics has great potential to provide for improvements in the effort to achieve real-time adaptive security. This implies an architecture which can detect security breaches and in real-time record and analyse traffic logs in a forensically sound manner, provide corrective feedback to security devices and attempt to trace back to the source of the attack. In addressing computer security and forensic analysis from a real-time perspective, this paper recognises that some of these processes already exist, but proposes methods whereby the ongoing damage and potential risk to critical infrastructure can be reduced. This requires the implementation of a highly integrated approach to security and forensics such that they can inter-work in real-time in order to address the significant security issues which currently face the industry.
---
paper_title: Decentralized multi-dimensional alert correlation for collaborative intrusion detection

paper_content:

The growth in coordinated network attacks such as scans, worms and distributed denial-of-service (DDoS) attacks is a profound threat to the security of the Internet. Collaborative intrusion detection systems (CIDSs) have the potential to detect these attacks, by enabling all the participating intrusion detection systems (IDSs) to share suspicious intelligence with each other to form a global view of the current security threats. Current correlation algorithms in CIDSs are either too simple to capture the important characteristics of attacks, or too computationally expensive to detect attacks in a timely manner. We propose a decentralized, multi-dimensional alert correlation algorithm for CIDSs to address these challenges. A multi-dimensional alert clustering algorithm is used to extract the significant intrusion patterns from raw intrusion alerts. A two-stage correlation algorithm is used, which first clusters alerts locally at each IDS, before reporting significant alert patterns to a global correlation stage. We introduce a probabilistic approach to decide when a pattern at the local stage is sufficiently significant to warrant correlation at the global stage. We then implement the proposed two-stage correlation algorithm in a fully distributed CIDS. Our experiments on a large real-world intrusion data set show that our approach can achieve a significant reduction in the number of alert messages generated by the local correlation stage with negligible false negatives compared to a centralized scheme. The proposed probabilistic threshold approach gains a significant improvement in detection accuracy in a stealthy attack scenario, compared to a naive scheme that uses the same threshold at the local and global stages. A large scale experiment on PlanetLab shows that our decentralized architecture is significantly more efficient than a centralized approach in terms of the time required to correlate alerts.
---
paper_title: Examining intrusion prevention system events from worldwide networks

paper_content:

We report preliminary results on analyzing a large dataset of over 35 billion alerts recorded over a 5 year period by Hewlett-Packard (HP) TippingPoint Intrusion Prevention System (IPS) devices located in over 1,000 customer networks worldwide. This dataset provides a rich view into the nature of attacks, both external and internal, across diverse networks. This paper presents our initial findings. For example, (i) while most customers are among the early victims of only a handful of attacks, a few customers are early victims of a large number of attacks, (ii) vendor vulnerability disclosures sometimes lead to a surge in exploit attempts, and (iii) even after a decade, some worms such as Slammer show very significant spikes in their activity and infection rates.
---
paper_title: A Flexible and Efficient Alert Correlation Platform for Distributed IDS

paper_content:

Intrusion Detection Systems (IDS) have been widely deployed in practice for detecting malicious behavior on network communication and hosts. The problem of false-positive alerts is a popular existing problem for most of IDS approaches. The solution to address this problem is correlation and clustering of alerts. To meet the practical requirements, this process needs to be finished as soon as possible, which is a challenging task as the amount of alerts produced in large scale deployments of distributed IDS is significantly high. We identify the data storage and processing algorithms to be the most important factors influencing the performance of clustering and correlation. We propose and implement the utilization of memory-supported algorithms and a column-oriented database for correlation and clustering in an extensible IDS correlation platform. The utilization of the column-oriented database, an In-Memory Alert Storage, and memory-based index tables leads to significant improvements on the performance. Different types of correlation modules can be integrated and compared on this platform. A plugin concept for Receivers provides flexible integration of various sensors and additional IDS management systems. The platform can be distributed over multiple processing units to share memory and processing power. A standardized interface is designed to provide a unified view of result reports for end users. The efficiency of the proposed platform is tested by practical experiments with several alert storage approaches, different simple algorithms, as well as local and distributed deployment.
---

Title: Intrusion Detection and Big Heterogeneous Data: A Survey

Section 1: Introduction
Description 1: Introduce the importance of Intrusion Detection and its evolution, highlight the challenges posed by heterogeneous data sources and Big Data.

Section 2: Intrusion detection and big data background
Description 2: Provide general background on Big Data and the challenges it presents for Intrusion Detection, including the traditional 3Vs (Volume, Velocity, Variety) and additional considerations like Veracity and Value.

Section 3: Big Heterogeneous Data definitions
Description 3: Define Big Heterogeneous Data and explain how it manifests in Intrusion Detection systems, providing illustrative equations and scenarios.

Section 4: Important considerations for intrusion detection
Description 4: Discuss how a comprehensive security monitoring system can enhance Intrusion Detection, and address relevant challenges, including historical context and economic implications.

Section 5: Challenges of machine learning in cybersecurity
Description 5: Address issues encountered with Intrusion Detection data sets and feature selection, emphasizing the importance of data stability and the rapid changes in the cybersecurity landscape.

Section 6: Intrusion detection data set challenges
Description 6: Examine the problems with existing public Intrusion Detection data sets and the need for up-to-date, relevant, and diverse data sets for research.

Section 7: Intrusion detection and feature selection opportunities
Description 7: Explore the role of feature selection in Intrusion Detection, its benefits, and the challenges associated with dynamically changing data and evolving threats.

Section 8: Using Hadoop to address big data challenges for intrusion detection
Description 8: Discuss the application of Hadoop and other Big Data technologies in managing the storage and processing demands of Intrusion Detection data.

Section 9: Security data across heterogeneous sources
Description 9: Explain techniques and architectures for accommodating diverse heterogeneous sources in Intrusion Detection and their implications on situational awareness.

Section 10: Enhancing situational awareness in cyberspace with data fusion
Description 10: Provide insights into data fusion techniques to improve situational awareness and their application in cybersecurity, borrowed from military strategies.

Section 11: A sampling of various heterogeneous intrusion detection architectures
Description 11: Present a selection of proposed Intrusion Detection architectures from academic studies that accommodate diverse heterogeneous sources.

Section 12: Security information and event management systems
Description 12: Provide an overview of SIEM systems, their functionalities, and their differences from traditional IDS solutions, focusing on their ability to handle diverse data sources.

Section 13: Big heterogeneous data for intrusion detection
Description 13: Describe Big Heterogeneous Data in terms of input and output categories, detailing the types of data involved and the Big Data challenges they present.

Section 14: Big Heterogeneous input data
Description 14: Further categorize Big Heterogeneous Input Data into traditional cyberspace data and industrial data, discussing specific challenges associated with each.

Section 15: Big Heterogeneous cyberspace data
Description 15: Explore the challenges posed by Big Data in traditional cyberspace input sources such as network and host layer data, and relevant case studies.

Section 16: Big Heterogeneous industrial data
Description 16: Discuss Big Data challenges in industrial and utility operations, their evolution, and the integration with cyberspace for improved situational awareness.

Section 17: Big heterogeneous output data
Description 17: Address the heterogeneity of output Big Data for Intrusion Detection, categorized into archival security data and alert data, and their respective challenges.

Section 18: Big archival security data
Description 18: Discuss the importance and challenges of long-term storage of security data for analytics and forensic purposes, including data mining and traceback capabilities.

Section 19: Big alert data
Description 19: Examine the challenges posed by generating, processing, and correlating alerts, and the techniques used to manage alert floods and improve detection accuracy.

Section 20: Discussion
Description 20: Summarize the main findings, highlight future research directions, and emphasize the need for more comprehensive data sets and technological advancements in Intrusion Detection.

Section 21: Conclusion
Description 21: Conclude by emphasizing the need for a diverse, heterogeneous approach to Intrusion Detection and the ongoing evolution required to address Big Data challenges effectively.