Anomaly Detection and Mitigation at Internet Scale: A Survey
---
paper_title: Botnet: Classification, Attacks, Detection, Tracing, and Preventive Measures

paper_content:

Botnets become widespread in wired and wireless networks, whereas the relevant research is still in the initial stage. In this paper, a survey of botnets is provided. We first discuss fundamental concepts of botnets, including formation and exploitation, lifecycle, and two major kinds of topologies. Several related attacks, detection, tracing, and countermeasures, are then introduced, followed by recent research work and possible future challenges.
---
paper_title: BotTrack : Tracking Botnets using NetFlow and PageRank

paper_content:

With large scale botnets emerging as one of the major current threats, the automatic detection of botnet traffic is of high importance for service providers and large campus network monitoring. Faced with high speed network connections, detecting botnets must be efficient and accurate. This paper proposes a novel approach for this task, where NetFlow related data is correlated and a host dependency model is leveraged for advanced data mining purposes. We extend the popular linkage analysis algorithm PageRank [27] with an additional clustering process in order to efficiently detect stealthy botnets using peer-to-peer communication infrastructures and not exhibiting large volumes of traffic. The key conceptual component in our approach is to analyze communication behavioral patterns and to infer potential botnet activities.
---
paper_title: Disclosure: detecting botnet command and control servers through large-scale NetFlow analysis

paper_content:

Botnets continue to be a significant problem on the Internet. Accordingly, a great deal of research has focused on methods for detecting and mitigating the effects of botnets. Two of the primary factors preventing the development of effective large-scale, wide-area botnet detection systems are seemingly contradictory. On the one hand, technical and administrative restrictions result in a general unavailability of raw network data that would facilitate botnet detection on a large scale. On the other hand, were this data available, real-time processing at that scale would be a formidable challenge. In contrast to raw network data, NetFlow data is widely available. However, NetFlow data imposes several challenges for performing accurate botnet detection. In this paper, we present Disclosure, a large-scale, wide-area botnet detection system that incorporates a combination of novel techniques to overcome the challenges imposed by the use of NetFlow data. In particular, we identify several groups of features that allow Disclosure to reliably distinguish C&C channels from benign traffic using NetFlow records (i.e., flow sizes, client access patterns, and temporal behavior). To reduce Disclosure's false positive rate, we incorporate a number of external reputation scores into our system's detection procedure. Finally, we provide an extensive evaluation of Disclosure over two large, real-world networks. Our evaluation demonstrates that Disclosure is able to perform real-time detection of botnet C&C channels over datasets on the order of billions of flows per day.
---
paper_title: RepCIDN: A Reputation-based Collaborative Intrusion Detection Network to Lessen the Impact of Malicious Alarms

paper_content:

Distributed and coordinated attacks in computer networks are causing considerable economic losses worldwide in recent years. This is mainly due to the transition of attackers’ operational patterns towards a more sophisticated and more global behavior. This fact is leading current intrusion detection systems to be more likely to generate false alarms. In this context, this paper describes the design of a collaborative intrusion detection network (CIDN) that is capable of building and sharing collective knowledge about isolated alarms in order to efficiently and accurately detect distributed attacks. It has been also strengthened with a reputation mechanism aimed to improve the detection coverage by dropping false or bogus alarms that arise from malicious or misbehaving nodes. This model will enable a CIDN to detect malicious behaviors according to the trustworthiness of the alarm issuers, calculated from previous interactions with the system. Experimental results will finally demonstrate how entities are gradually isolated as their behavior worsens throughout the time.
---
paper_title: BotTrack : Tracking Botnets using NetFlow and PageRank

paper_content:

With large scale botnets emerging as one of the major current threats, the automatic detection of botnet traffic is of high importance for service providers and large campus network monitoring. Faced with high speed network connections, detecting botnets must be efficient and accurate. This paper proposes a novel approach for this task, where NetFlow related data is correlated and a host dependency model is leveraged for advanced data mining purposes. We extend the popular linkage analysis algorithm PageRank [27] with an additional clustering process in order to efficiently detect stealthy botnets using peer-to-peer communication infrastructures and not exhibiting large volumes of traffic. The key conceptual component in our approach is to analyze communication behavioral patterns and to infer potential botnet activities.
---
paper_title: Disclosure: detecting botnet command and control servers through large-scale NetFlow analysis

paper_content:

Botnets continue to be a significant problem on the Internet. Accordingly, a great deal of research has focused on methods for detecting and mitigating the effects of botnets. Two of the primary factors preventing the development of effective large-scale, wide-area botnet detection systems are seemingly contradictory. On the one hand, technical and administrative restrictions result in a general unavailability of raw network data that would facilitate botnet detection on a large scale. On the other hand, were this data available, real-time processing at that scale would be a formidable challenge. In contrast to raw network data, NetFlow data is widely available. However, NetFlow data imposes several challenges for performing accurate botnet detection. In this paper, we present Disclosure, a large-scale, wide-area botnet detection system that incorporates a combination of novel techniques to overcome the challenges imposed by the use of NetFlow data. In particular, we identify several groups of features that allow Disclosure to reliably distinguish C&C channels from benign traffic using NetFlow records (i.e., flow sizes, client access patterns, and temporal behavior). To reduce Disclosure's false positive rate, we incorporate a number of external reputation scores into our system's detection procedure. Finally, we provide an extensive evaluation of Disclosure over two large, real-world networks. Our evaluation demonstrates that Disclosure is able to perform real-time detection of botnet C&C channels over datasets on the order of billions of flows per day.
---
paper_title: Operational Experiences with Anomaly Detection in Backbone Networks

paper_content:

Although network security is a crucial aspect for network operators, there are still very few works that have examined the anomalies present in large backbone networks and evaluated the performance of existing anomaly detection solutions in operational environments. The objective of this work is to fill this gap by reporting hands-on experience in the evaluation and deployment of an anomaly detection solution for the GEANT backbone network. During this process, we analyzed three different commercial tools for anomaly detection and then deployed one of them for several months in the 18 points-of-presence of GEANT. We first explain the general requirements that an anomaly detection system should satisfy from the point of view of a network operator. Afterwards, we describe the evaluation of the tools and present a study of the anomalies found in a continental backbone network after operationally using the finally deployed tool for half a year. We think that this first hand information can be of great interest to both professionals and researchers working on network security and can also guide future research towards more practical problems faced by network operators.
---

Title: Anomaly Detection and Mitigation at Internet Scale: A Survey

Section 1: Introduction
Description 1: Introduce the necessity and challenges of anomaly detection and mitigation in the internet landscape, detailing the different methodologies and scopes.

Section 2: Survey Description
Description 2: Describe the setup, methodology, and population of the survey aimed at ISPs and network operators.

Section 3: Result Set Analysis and Evaluation
Description 3: Present and evaluate the main results of the survey, divided into multiple sub-sections about compliance, threat awareness, data availability, and tools.

Section 4: Compliance to Security Standards and Frameworks
Description 4: Discuss the extent to which ISPs comply with established security standards and frameworks.

Section 5: Attacks and Threats
Description 5: Highlight the sources of information used by ISPs to stay aware of threats and detail the common attacks detected.

Section 6: Data and Tools
Description 6: Examine the types of data available at ISPs for anomaly detection and the tools they employ.

Section 7: Mitigation and Reaction
Description 7: Explore the current state of mitigation and reaction processes, detailing the time to respond and the measures in place.

Section 8: Role of ISP and IXP
Description 8: Assess the perceived role of ISPs and IXPs in network attack detection and mitigation.

Section 9: Conclusion and Future Work
Description 9: Summarize the findings of the survey and discuss the implications for future research and development in network-based anomaly detection.