A survey of techniques for internet traffic classification using machine learning
---
paper_title: Accurate, scalable in-network identification of p2p traffic using application signatures

paper_content:

The ability to accurately identify the network trafc associated with different P2P applications is important to a broad range of network operations including application-specic trafc engineering, capacity planning, provisioning, service differentiation, etc. However, traditional trafc to higher-level application mapping techniques such as default server TCP or UDP network-port based disambiguation is highly inaccurate for some P2P applications. In this paper, we provide an efcient approach for identifying the P2P application trafc through application level signatures. We rst identify the application level signatures by examining some available documentations, and packet-level traces. We then utilize the identied signatures to develop online lters that can efciently and accurately track the P2P trafc even on high-speed network links. We examine the performance of our application-level identication approach using ve popular P2P protocols. Our measurements show that our technique achieves less than false positive and false negative ratios in most cases. We also show that our approach only requires the examination of the very rst few packets (less than packets) to identify a P2P connection, which makes our approach highly scalable. Our technique can signicantly improve the P2P trafc volume estimates over what pure network port based approaches provide. For instance, we were able to identify times as much trafc for the popular Kazaa P2P protocol, compared to the traditional port-based approach.
---
paper_title: An Architecture for Automated Network Control of QoS over Consumer Broadband Links

paper_content:

This paper proposes a novel architecture for meeting Quality of Service (QoS) requirements of real-time traffic across consumer broadband links. In our approach the responsibility of QoS signalling is moved away from the application to the network. Network servers automatically identify traffic that might benefit from QoS and then trigger the provisioning of QoS by signalling network elements such as access routers. This approach removes the need for the application to signal to the network its explicit QoS requirements, making applications easier to develop and more portable. It also enables QoS provision for legacy applications for which there is limited opportunity to include explicit end-host signalling protocols. The paper develops the architecture required to realize the approach and discusses the underlying techniques.
---
paper_title: Beyond technology: the missing pieces for QoS success

paper_content:

Years of research on QoS architectures for IP networks have delivered sophisticated proposals, which have nevertheless not found broad commercial use. The reasons are not lack of technical soundness or insurmountable technological complexity, but insufficient attention to other, non-QoS-specific matters. First among them is the lack of a commercialization model for the Internet together with the necessary accounting and charging architecture. Another crucial issue is the assurance of end-to-end QoS coherence in the face of multiple intervening parties (network and content providers, users). Furthermore, the practical requirements imposed by those parties to any successful QoS architecture have not been fully taken into account: Ease of management, simplicity and measurable guarantees are some of the main ones. In this paper, the overall constraints on and conditions for the successful deployment of QoS in IP networks are analyzed and some possible directions explored.
---
paper_title: Integrated Services in the Internet Architecture: an Overview

paper_content:

This memo discusses a proposed extension to the Internet architecture and protocols to provide integrated services, i.e., to support real- time as well as the current non-real-time service of IP. This extension is necessary to meet the growing need for real-time service for a variety of new applications, including teleconferencing, remote seminars, telescience, and distributed simulation.
---
paper_title: An Architecture for Differentiated Service

paper_content:

This document defines an architecture for implementing scalable service differentiation in the Internet. This architecture achieves scalability by aggregating traffic classification state which is conveyed by means of IP-layer packet marking using the DS field [DSFIELD]. Packets are classified and marked to receive a particular per-hop forwarding behavior on nodes along their path. Sophisticated classification, marking, policing, and shaping operations need only be implemented at network boundaries or hosts. Network resources are allocated to traffic streams by service provisioning policies which govern how traffic is marked and conditioned upon entry to a differentiated services-capable network, and how that traffic is forwarded within that network. A wide variety of services can be implemented on top of these building blocks.
---
paper_title: An Architecture for Automated Network Control of QoS over Consumer Broadband Links

paper_content:

This paper proposes a novel architecture for meeting Quality of Service (QoS) requirements of real-time traffic across consumer broadband links. In our approach the responsibility of QoS signalling is moved away from the application to the network. Network servers automatically identify traffic that might benefit from QoS and then trigger the provisioning of QoS by signalling network elements such as access routers. This approach removes the need for the application to signal to the network its explicit QoS requirements, making applications easier to develop and more portable. It also enables QoS provision for legacy applications for which there is limited opportunity to include explicit end-host signalling protocols. The paper develops the architecture required to realize the approach and discusses the underlying techniques.
---
paper_title: Internet traffic classification using bayesian analysis techniques

paper_content:

Accurate traffic classification is of fundamental importance to numerous other network activities, from security monitoring to accounting, and from Quality of Service to providing operators with useful forecasts for long-term provisioning. We apply a Naive Bayes estimator to categorize traffic by application. Uniquely, our work capitalizes on hand-classified network data, using it as input to a supervised Naive Bayes estimator. In this paper we illustrate the high level of accuracy achievable with the \Naive Bayes estimator. We further illustrate the improved accuracy of refined variants of this estimator.Our results indicate that with the simplest of Naive Bayes estimator we are able to achieve about 65% accuracy on per-flow classification and with two powerful refinements we can improve this value to better than 95%; this is a vast improvement over traditional techniques that achieve 50--70%. While our technique uses training data, with categories derived from packet-content, all of our training and testing was done using header-derived discriminators. We emphasize this as a powerful aspect of our approach: using samples of well-known traffic to allow the categorization of traffic using commonly available information alone.
---
paper_title: Byte me: a case for byte accuracy in traffic classification

paper_content:

Numerous network traffic classification approaches have recently been proposed. In general, these approaches have focused on correctly identifying a high percentage of total flows. However, on the Internet a small number of "elephant" flows contribute a significant amount of the traffic volume. In addition, some application types like Peer-to-Peer (P2P) and FTP contribute more elephant flows than other applications types like Chat. In this opinion piece, we discuss how evaluating a classifier on flow accuracy alone can bias the classification results. By not giving special attention to these traffic classes and their elephant flows in the evaluation of traffic classification approaches we might obtain significantly different performance when these approaches are deployed in operational networks for typical traffic classification tasks such as traffic shaping. We argue that byte accuracy must also be used when evaluating the accuracy of traffic classification algorithms.
---
paper_title: Toward the accurate identification of network applications

paper_content:

Well-known port numbers can no longer be used to reliably identify network applications. There is a variety of new Internet applications that either do not use well-known port numbers or use other protocols, such as HTTP, as wrappers in order to go through firewalls without being blocked. One consequence of this is that a simple inspection of the port numbers used by flows may lead to the inaccurate classification of network traffic. In this work, we look at these inaccuracies in detail. Using a full payload packet trace collected from an Internet site we attempt to identify the types of errors that may result from port-based classification and quantify them for the specific trace under study. To address this question we devise a classification methodology that relies on the full packet payload. We describe the building blocks of this methodology and elaborate on the complications that arise in that context. A classification technique approaching 100% accuracy proves to be a labor-intensive process that needs to test flow-characteristics against multiple classification criteria in order to gain sufficient confidence in the nature of the causal application. Nevertheless, the benefits gained from a content-based classification approach are evident. We are capable of accurately classifying what would be otherwise classified as unknown as well as identifying traffic flows that could otherwise be classified incorrectly. Our work opens up multiple research issues that we intend to address in future work.
---
paper_title: Class-of-service mapping for QoS: a statistical signature-based approach to IP traffic classification

paper_content:

The ability to provide different Quality of Service (QoS) guarantees to traffic from different applications is a highly desired feature for many IP network operators, particularly for enterprise networks. Although various mechanisms exist for providing QoS in the network, QoS is yet to be widely deployed. We believe that a key factor holding back widespread QoS adoption is the absence of suitable methodologies/processes for appropriately mapping the traffic from different applications to different QoS classes. This is a challenging task, because many enterprise network operators who are interested in QoS do not know all the applications running on their network, and furthermore, over recent years port-based application classification has become problematic. We argue that measurement based automated Class of Service (CoS) mapping is an important practical problem that needs to be studied. In this paper we describe the requirements and associated challenges, and outline a solution framework for measurement based classification of traffic for QoS based on statistical application signatures. In our approach the signatures are chosen in such as way as to make them insensitive to the particular application layer protocol, but rather to determine the way in which an application is used -- for instance is it used interactively, or for bulk-data transport. The resulting application signature can then be used to derive the network layer signatures required to determine the CoS class for individual IP datagrams. Our evaluations using traffic traces from a variety of network locations, demonstrate the feasibility and potential of the approach.
---
paper_title: Accurate, scalable in-network identification of p2p traffic using application signatures

paper_content:

The ability to accurately identify the network trafc associated with different P2P applications is important to a broad range of network operations including application-specic trafc engineering, capacity planning, provisioning, service differentiation, etc. However, traditional trafc to higher-level application mapping techniques such as default server TCP or UDP network-port based disambiguation is highly inaccurate for some P2P applications. In this paper, we provide an efcient approach for identifying the P2P application trafc through application level signatures. We rst identify the application level signatures by examining some available documentations, and packet-level traces. We then utilize the identied signatures to develop online lters that can efciently and accurately track the P2P trafc even on high-speed network links. We examine the performance of our application-level identication approach using ve popular P2P protocols. Our measurements show that our technique achieves less than false positive and false negative ratios in most cases. We also show that our approach only requires the examination of the very rst few packets (less than packets) to identify a P2P connection, which makes our approach highly scalable. Our technique can signicantly improve the P2P trafc volume estimates over what pure network port based approaches provide. For instance, we were able to identify times as much trafc for the popular Kazaa P2P protocol, compared to the traditional port-based approach.
---
paper_title: A Longitudinal Study of P2P Traffic Classification

paper_content:

This paper focuses on network traffic measurement of Peer-to- Peer (P2P) applications on the Internet. P2P applications supposedly constitute a substantial proportion of today's Internet traffic. However, current P2P applications use several obfuscation techniques, including dynamic port numbers, port hopping, HTTP masquerading, chunked file transfers, and encrypted payloads. As P2P applications continue to evolve, robust and effective methods are needed for P2P traffic identification. The paper compares three methods to classify P2P applications: port-based classification, application-layer signatures, and transport-layer analysis. The study uses empirical network traces collected from the University of Calgary Internet connection for the past 2 years. The results show that port-based analysis is ineffective, being unable to identify 30%-70% of today's Internet traffic. Application signatures are accurate, but may not be possible for legal or technical reasons. The transport-layer method seems promising, providing a robust means to assess aggregate P2P traffic. The latter method suggests that 30%-70% of the campus Internet traffic for the past year was P2P.
---
paper_title: A synthetic traffic model for Quake3

paper_content:

This paper presents our development of a synthetic traffic model for the interactive online computer game Quake3. The goal is a traffic model that can be used by researchers and Internet Service Provider engineers to estimate the potential future impact of Quake3 traffic over IP networks. We developed our ns2 simulation model for Quake3's IP traffic by running live network experiments and characterising the observed packet length, packet inter-arrival times, and data rates (in packets- and bits- per second). Our observations are documented in this paper, and we expect our traffic model will assist network planners who wish to better support real-time game traffic.
---
paper_title: Empirically derived analytic models of wide-area TCP connections

paper_content:

Analyzes 3 million TCP connections that occurred during 15 wide-area traffic traces. The traces were gathered at five "stub" networks and two internetwork gateways, providing a diverse look at wide-area traffic. The author derives analytic models describing the random variables associated with TELNET, NNTP, SMTP, and FTP connections. To assess these models the author presents a quantitative methodology for comparing their effectiveness with that of empirical models such as Tcplib [Danzig and Jamin, 1991]. The methodology also allows to determine which random variables show significant variation from site to site, over time, or between stub networks and internetwork gateways. Overall the author finds that the analytic models provide good descriptions, and generally model the various distributions as well as empirical models. >
---
paper_title: An analysis of Internet chat systems

paper_content:

In our quest to better understand network traffic dynamics, we examine Internet chat systems. Although chat as an application does not contribute huge amounts of traffic, chat systems are known to be habit-forming. This implies that catering to such users can be a promising way of attracting them, especially in low bandwidth environments such as wireless networks. Unfortunately there is no common protocol base for chat systems. Rather there are a multitude of protocol variants whose specifications, with some exceptions, such as IRC and ICQ, are unavailable or ill defined. In addition, chat systems are often layered on top of other application protocols like HTTP. Therefore there is no simple way of even identifying chat traffic. In this paper we show how to separate chat traffic from other Internet traffic and present the results of an extensive validation of our methodology. Using our methodology we gather a week long trace of all chat traffic that crosses a 155 Mbit/s link from the Saarland University to the Internet and present an initial characterization.
---
paper_title: Data mining: practical machine learning tools and techniques with Java implementations

paper_content:

1. What's It All About? 2. Input: Concepts, Instances, Attributes 3. Output: Knowledge Representation 4. Algorithms: The Basic Methods 5. Credibility: Evaluating What's Been Learned 6. Implementations: Real Machine Learning Schemes 7. Moving On: Engineering The Input And Output 8. Nuts And Bolts: Machine Learning Algorithms In Java 9. Looking Forward
---
paper_title: Netman: a learning network traffic controller

paper_content:

One of the goals of Machine Learning is the production of software that can improve itself. Such software can learn from experience and adapt to changing situations and requirements. In addition, such software can refine its knowledge-base, perhaps leading to a level of expertise beyond that of human experts. This paper describes NETMAN, a knowledge-based program that uses a machine learning technique, Knowledge-based Learning, in the domain of Network Traffic Control. NETMAN's task is to maximize call completion in a circuit-switched telecommunications network. NETMAN learns from its own experiences and by observing the actions of other agents. NETMAN is one of the components of ILS ( I ntegrated L earning S ystem), which contains implementations of several learning paradigms working together to improve problem-solving performance. NETMAN combines two machine learning paradigms: Explanation-Based Learning and Empirical Learning .
---
paper_title: Concept formation knowledge and experience in unsupervised learning

paper_content:

Concept formation lies at the center of learning and cognition. Unlike much work in machine learning and cognitive psychology, research on this topic focuses on the unsupervised and incremental acquisition of conceptual knowledge. Recent work on concept formation addresses a number of important issues. Foremost among these are the principles of similarity that guide concept learning and retrieval in human and machine, including the contribution of surface features, goals, and `deep' features. Another active area of research explores mechanisms for efficiently reorganizing memory in response to the ongoing experiences that confront intelligent agents. Finally, methods for concept formation play an increasing role in work on problem solving and planning, developmental psychology, engineering applications, and constructive induction. This book brings together results on concept formation from cognitive psychology and machine learning, including explanation-based and inductive approaches. Chapters from these differing perspectives are intermingled to highlight the commonality of their research agendas. In addition to cognitive scientists and AI researchers, the book will interest data analysts involved in clustering, philosophers concerned with the nature and origin of concepts, and any researcher dealing with issues of similarity, memory organization, and problem solving.
---
paper_title: Objective Criteria for the Evaluation of Clustering Methods

paper_content:

Abstract Many intuitively appealing methods have been suggested for clustering data, however, interpretation of their results has been hindered by the lack of objective criteria. This article proposes several criteria which isolate specific aspects of the performance of a method, such as its retrieval of inherent structure, its sensitivity to resampling and the stability of its results in the light of new data. These criteria depend on a measure of similarity between two different clusterings of the same set of data; the measure essentially considers how each pair of data points is assigned in each clustering.
---
paper_title: Clustering validity checking methods: part II

paper_content:

Clustering results validation is an important topic in the context of pattern recognition. We review approaches and systems in this context. In the first part of this paper we presented clustering validity checking approaches based on internal and external criteria. In the second, current part, we present a review of clustering validity approaches based on relative criteria. Also we discuss the results of an experimental study based on widely known validity indices. Finally the paper illustrates the issues that are under-addressed by the recent approaches and proposes the research directions in the field.
---
paper_title: Data mining: practical machine learning tools and techniques with Java implementations

paper_content:

1. What's It All About? 2. Input: Concepts, Instances, Attributes 3. Output: Knowledge Representation 4. Algorithms: The Basic Methods 5. Credibility: Evaluating What's Been Learned 6. Implementations: Real Machine Learning Schemes 7. Moving On: Engineering The Input And Output 8. Nuts And Bolts: Machine Learning Algorithms In Java 9. Looking Forward
---
paper_title: Survey of clustering algorithms

paper_content:

Data analysis plays an indispensable role for understanding various phenomena. Cluster analysis, primitive exploration with little or no prior knowledge, consists of research developed across a wide variety of communities. The diversity, on one hand, equips us with many tools. On the other hand, the profusion of options causes confusion. We survey clustering algorithms for data sets appearing in statistics, computer science, and machine learning, and illustrate their applications in some benchmark data sets, the traveling salesman problem, and bioinformatics, a new field attracting intensive efforts. Several tightly related topics, proximity measure, and cluster validation, are also discussed.
---
paper_title: Cluster validity methods: part I

paper_content:

Clustering is an unsupervised process since there are no predefined classes and no examples that would indicate grouping properties in the data set. The majority of the clustering algorithms behave differently depending on the features of the data set and the initial assumptions for defining groups. Therefore, in most applications the resulting clustering scheme requires some sort of evaluation as regards its validity. Evaluating and assessing the results of a clustering algorithm is the main subject of cluster validity. In this paper we present a review of the clustering validity and methods. More specifically, Part I of the paper discusses the cluster validity approaches based on external and internal criteria.
---
paper_title: Shape-based recognition of wiry objects

paper_content:

We present an approach to the recognition of complex-shaped objects in cluttered environments based on edge information. We first use example images of a target object in typical environments to train a classifier cascade that determines whether edge pixels in an image belong to an instance of the desired object or the clutter. Presented with a novel image, we use the cascade to discard clutter edge pixels and group the object edge pixels into overall detections of the object. The features used for the edge pixel classification are localized, sparse edge density operations. Experiments validate the effectiveness of the technique for recognition of a set of complex objects in a variety of cluttered indoor scenes under arbitrary out-of-image-plane rotation. Furthermore, our experiments suggest that the technique is robust to variations between training and testing environments and is efficient at runtime.
---
paper_title: Internet Traffic Classification for Scalable QOS Provision

paper_content:

A new scheme that classifies the Internet traffic according to their application types for scalable QoS provision is proposed in this work. The traditional port-based classification method does not yield satisfactory performance, since the same port can be shared by multiple applications. Furthermore, asymmetric routing and errors of modern measurement tools such as PCF and NetFlow degrades the classification performance. To address these issues, the proposed classification process consists of two steps: feature selection and classification. Candidate features that can be obtained easily by ISP are examined. Then, we perform feature reduction so as to balance the performance and complexity. As to classification, the REPTree and the bagging schemes are adopted and compared. It is demonstrated by simulations with real data that the proposed classification scheme outperforms existing techniques.
---
paper_title: Traffic classification using clustering algorithms

paper_content:

Classification of network traffic using port-based or payload-based analysis is becoming increasingly difficult with many peer-to-peer (P2P) applications using dynamic port numbers, masquerading techniques, and encryption to avoid detection. An alternative approach is to classify traffic by exploiting the distinctive characteristics of applications when they communicate on a network. We pursue this latter approach and demonstrate how cluster analysis can be used to effectively identify groups of traffic that are similar using only transport layer statistics. Our work considers two unsupervised clustering algorithms, namely K-Means and DBSCAN, that have previously not been used for network traffic classification. We evaluate these two algorithms and compare them to the previously used AutoClass algorithm, using empirical Internet traces. The experimental results show that both K-Means and DBSCAN work very well and much more quickly then AutoClass. Our results indicate that although DBSCAN has lower accuracy compared to K-Means and AutoClass, DBSCAN produces better clusters.
---
paper_title: Identifying and discriminating between web and peer-to-peer traffic in the network core

paper_content:

Traffic classification is the ability to identify and categorize network traffic by application type. In this paper, we consider the problem of traffic classification in the network core.Classification at the core is challenging because only partial information about the flows and their contributors is available. We address this problem by developing a framework that can classify a flow using only unidirectional flow information. We evaluated this approach using recent packet traces that we collected and pre-classified to establish a "base truth". From our evaluation, we find that flow statistics for the server-to-client direction of a TCP connection provide greater classification accuracy than the flow statistics for the client-to-server direction. Because collection of the server-to-client flow statistics may not always be feasible, we developed and validated an algorithm that can estimate the missing statistics froma unidirectional packet trace.
---
paper_title: Automated traffic classification and application identification using machine learning

paper_content:

The dynamic classification and identification of network applications responsible for network traffic flows offers substantial benefits to a number of key areas in IP network engineering, management and surveillance. Currently such classifications rely on selected packet header fields (e.g. port numbers) or application layer protocol decoding. These methods have a number of shortfalls e.g. many applications can use unpredictable port numbers and protocol decoding requires a high amount of computing resources or is simply infeasible in case protocols are unknown or encrypted. We propose a novel method for traffic classification and application identification using an unsupervised machine learning technique. Flows are automatically classified based on statistical flow characteristics. We evaluate the efficiency of our approach using data from several traffic traces collected at different locations of the Internet. We use feature selection to find an optimal feature set and determine the influence of different features
---
paper_title: Traffic classification on the fly

paper_content:

The early detection of applications associated with TCP flows is an essential step for network security and traffic engineering. The classic way to identify flows, i.e. looking at port numbers, is not effective anymore. On the other hand, state-of-the-art techniques cannot determine the application before the end of the TCP flow. In this editorial, we propose a technique that relies on the observation of the first five packets of a TCP connection to identify the application. This result opens a range of new possibilities for online traffic classification.
---
paper_title: A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification

paper_content:

The identification of network applications through observation of associated packet traffic flows is vital to the areas of network management and surveillance. Currently popular methods such as port number and payload-based identification exhibit a number of shortfalls. An alternative is to use machine learning (ML) techniques and identify network applications based on per-flow statistics, derived from payload-independent features such as packet length and inter-arrival time distributions. The performance impact of feature set reduction, using Consistency-based and Correlation-based feature selection, is demonstrated on Naive Bayes, C4.5, Bayesian Network and Naive Bayes Tree algorithms. We then show that it is useful to differentiate algorithms based on computational performance rather than classification accuracy alone, as although classification accuracy between the algorithms is similar, computational performance can differ significantly.
---
paper_title: Internet traffic classification using bayesian analysis techniques

paper_content:

Accurate traffic classification is of fundamental importance to numerous other network activities, from security monitoring to accounting, and from Quality of Service to providing operators with useful forecasts for long-term provisioning. We apply a Naive Bayes estimator to categorize traffic by application. Uniquely, our work capitalizes on hand-classified network data, using it as input to a supervised Naive Bayes estimator. In this paper we illustrate the high level of accuracy achievable with the \Naive Bayes estimator. We further illustrate the improved accuracy of refined variants of this estimator.Our results indicate that with the simplest of Naive Bayes estimator we are able to achieve about 65% accuracy on per-flow classification and with two powerful refinements we can improve this value to better than 95%; this is a vast improvement over traditional techniques that achieve 50--70%. While our technique uses training data, with categories derived from packet-content, all of our training and testing was done using header-derived discriminators. We emphasize this as a powerful aspect of our approach: using samples of well-known traffic to allow the categorization of traffic using commonly available information alone.
---
paper_title: Traffic classification through simple statistical fingerprinting

paper_content:

The classification of IP ows according to the application that generated them is at the basis of any modern network management platform. However, classical techniques such as the ones based on the analysis of transport layer or application layer information are rapidly becoming ineffective. In this paper we present a ow classification mechanism based on three simple properties of the captured IP packets: their size, inter-arrival time and arrival order. Even though these quantities have already been used in the past to define classification techniques, our contribution is based on new structures called protocol fingerprints, which express such quantities in a compact and efficient way, and on a simple classification algorithm based on normalized thresholds. Although at a very early stage of development, the proposed technique is showing promising preliminary results from the classification of a reduced set of protocols.
---
paper_title: Automated traffic classification and application identification using machine learning

paper_content:

The dynamic classification and identification of network applications responsible for network traffic flows offers substantial benefits to a number of key areas in IP network engineering, management and surveillance. Currently such classifications rely on selected packet header fields (e.g. port numbers) or application layer protocol decoding. These methods have a number of shortfalls e.g. many applications can use unpredictable port numbers and protocol decoding requires a high amount of computing resources or is simply infeasible in case protocols are unknown or encrypted. We propose a novel method for traffic classification and application identification using an unsupervised machine learning technique. Flows are automatically classified based on statistical flow characteristics. We evaluate the efficiency of our approach using data from several traffic traces collected at different locations of the Internet. We use feature selection to find an optimal feature set and determine the influence of different features
---
paper_title: Training on multiple sub-flows to optimise the use of Machine Learning classifiers in real-world IP networks

paper_content:

Literature on the use of machine learning (ML) algorithms for classifying IP traffic has relied on full-flows or the first few packets of flows. In contrast, many real-world scenarios require a classification decision well before a flow has finished even if the flow's beginning is lost. This implies classification must be achieved using statistics derived from the most recent N packets taken at any arbitrary point in a flow's lifetime. We propose training the classifier on a combination of short sub-flows (extracted from full-flow examples of the target application's traffic). We demonstrate this optimisation using the Naive Bayes ML algorithm, and show that our approach results in excellent performance even when classification is initiated mid-way through a flow with windows as small as 25 packets long. We suggest future use of unsupervised ML algorithms to identify optimal sub-flows for training.
---
paper_title: Flow Clustering Using Machine Learning Techniques

paper_content:

Packet header traces are widely used in network analysis. Header traces are the aggregate of traffic from many concurrent applications. We present a methodology, based on machine learning, that can break the trace down into clusters of traffic where each cluster has different traffic characteristics. Typical clusters include bulk transfer, single and multiple transactions and interactive traffic, amongst others. The paper includes a description of the methodology, a visualisation of the attribute statistics that aids in recognising cluster types and a discussion of the stability and effectiveness of the methodology.
---
paper_title: Traffic classification on the fly

paper_content:

The early detection of applications associated with TCP flows is an essential step for network security and traffic engineering. The classic way to identify flows, i.e. looking at port numbers, is not effective anymore. On the other hand, state-of-the-art techniques cannot determine the application before the end of the TCP flow. In this editorial, we propose a technique that relies on the observation of the first five packets of a TCP connection to identify the application. This result opens a range of new possibilities for online traffic classification.
---
paper_title: Maximum likelihood from incomplete data via the EM algorithm

paper_content:

Vibratory power unit for vibrating conveyers and screens comprising an asynchronous polyphase motor, at least one pair of associated unbalanced masses disposed on the shaft of said motor, with the first mass of a pair of said unbalanced masses being rigidly fastened to said shaft and with said second mass of said pair being movably arranged relative to said first mass, means for controlling and regulating the conveying rate during conveyer operation by varying the rotational speed of said motor between predetermined minimum and maximum values, said second mass being movably outwardly by centrifugal force against the pressure of spring means, said spring means being prestressed in such a manner that said second mass is, at rotational motor speeds lower than said minimum speed, held in its initial position, and at motor speeds between said lower and upper values in positions which are radially offset with respect to the axis of said motor to an extent depending on the value of said rotational motor speed.
---
paper_title: Automated traffic classification and application identification using machine learning

paper_content:

The dynamic classification and identification of network applications responsible for network traffic flows offers substantial benefits to a number of key areas in IP network engineering, management and surveillance. Currently such classifications rely on selected packet header fields (e.g. port numbers) or application layer protocol decoding. These methods have a number of shortfalls e.g. many applications can use unpredictable port numbers and protocol decoding requires a high amount of computing resources or is simply infeasible in case protocols are unknown or encrypted. We propose a novel method for traffic classification and application identification using an unsupervised machine learning technique. Flows are automatically classified based on statistical flow characteristics. We evaluate the efficiency of our approach using data from several traffic traces collected at different locations of the Internet. We use feature selection to find an optimal feature set and determine the influence of different features
---
paper_title: Identifying and discriminating between web and peer-to-peer traffic in the network core

paper_content:

Traffic classification is the ability to identify and categorize network traffic by application type. In this paper, we consider the problem of traffic classification in the network core.Classification at the core is challenging because only partial information about the flows and their contributors is available. We address this problem by developing a framework that can classify a flow using only unidirectional flow information. We evaluated this approach using recent packet traces that we collected and pre-classified to establish a "base truth". From our evaluation, we find that flow statistics for the server-to-client direction of a TCP connection provide greater classification accuracy than the flow statistics for the client-to-server direction. Because collection of the server-to-client flow statistics may not always be feasible, we developed and validated an algorithm that can estimate the missing statistics froma unidirectional packet trace.
---
paper_title: Class-of-service mapping for QoS: a statistical signature-based approach to IP traffic classification

paper_content:

The ability to provide different Quality of Service (QoS) guarantees to traffic from different applications is a highly desired feature for many IP network operators, particularly for enterprise networks. Although various mechanisms exist for providing QoS in the network, QoS is yet to be widely deployed. We believe that a key factor holding back widespread QoS adoption is the absence of suitable methodologies/processes for appropriately mapping the traffic from different applications to different QoS classes. This is a challenging task, because many enterprise network operators who are interested in QoS do not know all the applications running on their network, and furthermore, over recent years port-based application classification has become problematic. We argue that measurement based automated Class of Service (CoS) mapping is an important practical problem that needs to be studied. In this paper we describe the requirements and associated challenges, and outline a solution framework for measurement based classification of traffic for QoS based on statistical application signatures. In our approach the signatures are chosen in such as way as to make them insensitive to the particular application layer protocol, but rather to determine the way in which an application is used -- for instance is it used interactively, or for bulk-data transport. The resulting application signature can then be used to derive the network layer signatures required to determine the CoS class for individual IP datagrams. Our evaluations using traffic traces from a variety of network locations, demonstrate the feasibility and potential of the approach.
---
paper_title: Traffic classification on the fly

paper_content:

The early detection of applications associated with TCP flows is an essential step for network security and traffic engineering. The classic way to identify flows, i.e. looking at port numbers, is not effective anymore. On the other hand, state-of-the-art techniques cannot determine the application before the end of the TCP flow. In this editorial, we propose a technique that relies on the observation of the first five packets of a TCP connection to identify the application. This result opens a range of new possibilities for online traffic classification.
---
paper_title: Internet traffic classification using bayesian analysis techniques

paper_content:

Accurate traffic classification is of fundamental importance to numerous other network activities, from security monitoring to accounting, and from Quality of Service to providing operators with useful forecasts for long-term provisioning. We apply a Naive Bayes estimator to categorize traffic by application. Uniquely, our work capitalizes on hand-classified network data, using it as input to a supervised Naive Bayes estimator. In this paper we illustrate the high level of accuracy achievable with the \Naive Bayes estimator. We further illustrate the improved accuracy of refined variants of this estimator.Our results indicate that with the simplest of Naive Bayes estimator we are able to achieve about 65% accuracy on per-flow classification and with two powerful refinements we can improve this value to better than 95%; this is a vast improvement over traditional techniques that achieve 50--70%. While our technique uses training data, with categories derived from packet-content, all of our training and testing was done using header-derived discriminators. We emphasize this as a powerful aspect of our approach: using samples of well-known traffic to allow the categorization of traffic using commonly available information alone.
---
paper_title: Bayesian Neural Networks for Internet Traffic Classification

paper_content:

Internet traffic identification is an important tool for network management. It allows operators to better predict future traffic matrices and demands, security personnel to detect anomalous behavior, and researchers to develop more realistic traffic models. We present here a traffic classifier that can achieve a high accuracy across a range of application types without any source or destination host-address or port information. We use supervised machine learning based on a Bayesian trained neural network. Though our technique uses training data with categories derived from packet content, training and testing were done using features derived from packet streams consisting of one or more packet headers. By providing classification without access to the contents of packets, our technique offers wider application than methods that require full packet/payloads for classification. This is a powerful advantage, using samples of classified traffic to permit the categorization of traffic based only upon commonly available information
---
paper_title: ACAS: automated construction of application signatures

paper_content:

An accurate mapping of traffic to applications is important for a broad range of network management and measurement tasks. Internet applications have traditionally been identified using well-known default server network-port numbers in the TCP or UDP headers. However this approach has become increasingly inaccurate. An alternate, more accurate technique is to use specific application-level features in the protocol exchange to guide the identification. Unfortunately deriving the signatures manually is very time consuming and difficult.In this paper, we explore automatically extracting application signatures from IP traffic payload content. In particular we apply three statistical machine learning algorithms to automatically identify signatures for a range of applications. The results indicate that this approach is highly accurate and scales to allow online application identification on high speed links. We also discovered that content signatures still work in the presence of encryption. In these cases we were able to derive content signature for unencrypted handshakes negotiating the encryption parameters of a particular connection.
---
paper_title: Training on multiple sub-flows to optimise the use of Machine Learning classifiers in real-world IP networks

paper_content:

Literature on the use of machine learning (ML) algorithms for classifying IP traffic has relied on full-flows or the first few packets of flows. In contrast, many real-world scenarios require a classification decision well before a flow has finished even if the flow's beginning is lost. This implies classification must be achieved using statistics derived from the most recent N packets taken at any arbitrary point in a flow's lifetime. We propose training the classifier on a combination of short sub-flows (extracted from full-flow examples of the target application's traffic). We demonstrate this optimisation using the Naive Bayes ML algorithm, and show that our approach results in excellent performance even when classification is initiated mid-way through a flow with windows as small as 25 packets long. We suggest future use of unsupervised ML algorithms to identify optimal sub-flows for training.
---
paper_title: Training on multiple sub-flows to optimise the use of Machine Learning classifiers in real-world IP networks

paper_content:

Literature on the use of machine learning (ML) algorithms for classifying IP traffic has relied on full-flows or the first few packets of flows. In contrast, many real-world scenarios require a classification decision well before a flow has finished even if the flow's beginning is lost. This implies classification must be achieved using statistics derived from the most recent N packets taken at any arbitrary point in a flow's lifetime. We propose training the classifier on a combination of short sub-flows (extracted from full-flow examples of the target application's traffic). We demonstrate this optimisation using the Naive Bayes ML algorithm, and show that our approach results in excellent performance even when classification is initiated mid-way through a flow with windows as small as 25 packets long. We suggest future use of unsupervised ML algorithms to identify optimal sub-flows for training.
---
paper_title: Traffic classification on the fly

paper_content:

The early detection of applications associated with TCP flows is an essential step for network security and traffic engineering. The classic way to identify flows, i.e. looking at port numbers, is not effective anymore. On the other hand, state-of-the-art techniques cannot determine the application before the end of the TCP flow. In this editorial, we propose a technique that relies on the observation of the first five packets of a TCP connection to identify the application. This result opens a range of new possibilities for online traffic classification.
---
paper_title: Internet Traffic Classification for Scalable QOS Provision

paper_content:

A new scheme that classifies the Internet traffic according to their application types for scalable QoS provision is proposed in this work. The traditional port-based classification method does not yield satisfactory performance, since the same port can be shared by multiple applications. Furthermore, asymmetric routing and errors of modern measurement tools such as PCF and NetFlow degrades the classification performance. To address these issues, the proposed classification process consists of two steps: feature selection and classification. Candidate features that can be obtained easily by ISP are examined. Then, we perform feature reduction so as to balance the performance and complexity. As to classification, the REPTree and the bagging schemes are adopted and compared. It is demonstrated by simulations with real data that the proposed classification scheme outperforms existing techniques.
---
paper_title: GA-Based Internet Traffic Classification Technique for QoS Provisioning

paper_content:

A fast and robust scheme that classifies Internet packets according to their application types is proposed in this work. The scheme is deployed at ISP for QoS provisioning, scalability and reliability. The proposed classification scheme consists of two steps: feature selection and classification. For feature selection, practical features are extracted using the modified multistage filter. By using the genetic algorithm (GA) and a variant of the wrapper method, we obtain two sets of features for comparison. As to classifiers, decision trees such as J48 and REPTree. The decision trees are trained with selected features from real traffics. The trained decision trees are compared with a classifier using the NBKE approach in terms of accuracy and robustness. It is demonstrated by simulation results that decision trees with features selected by GA gives the best performance. Finally, early classification with modified multistage filters is proposed to reduce collision errors for fast and robust performance.
---
paper_title: Traffic classification through simple statistical fingerprinting

paper_content:

The classification of IP ows according to the application that generated them is at the basis of any modern network management platform. However, classical techniques such as the ones based on the analysis of transport layer or application layer information are rapidly becoming ineffective. In this paper we present a ow classification mechanism based on three simple properties of the captured IP packets: their size, inter-arrival time and arrival order. Even though these quantities have already been used in the past to define classification techniques, our contribution is based on new structures called protocol fingerprints, which express such quantities in a compact and efficient way, and on a simple classification algorithm based on normalized thresholds. Although at a very early stage of development, the proposed technique is showing promising preliminary results from the classification of a reduced set of protocols.
---
paper_title: A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification

paper_content:

The identification of network applications through observation of associated packet traffic flows is vital to the areas of network management and surveillance. Currently popular methods such as port number and payload-based identification exhibit a number of shortfalls. An alternative is to use machine learning (ML) techniques and identify network applications based on per-flow statistics, derived from payload-independent features such as packet length and inter-arrival time distributions. The performance impact of feature set reduction, using Consistency-based and Correlation-based feature selection, is demonstrated on Naive Bayes, C4.5, Bayesian Network and Naive Bayes Tree algorithms. We then show that it is useful to differentiate algorithms based on computational performance rather than classification accuracy alone, as although classification accuracy between the algorithms is similar, computational performance can differ significantly.
---
paper_title: Traffic classification using clustering algorithms

paper_content:

Classification of network traffic using port-based or payload-based analysis is becoming increasingly difficult with many peer-to-peer (P2P) applications using dynamic port numbers, masquerading techniques, and encryption to avoid detection. An alternative approach is to classify traffic by exploiting the distinctive characteristics of applications when they communicate on a network. We pursue this latter approach and demonstrate how cluster analysis can be used to effectively identify groups of traffic that are similar using only transport layer statistics. Our work considers two unsupervised clustering algorithms, namely K-Means and DBSCAN, that have previously not been used for network traffic classification. We evaluate these two algorithms and compare them to the previously used AutoClass algorithm, using empirical Internet traces. The experimental results show that both K-Means and DBSCAN work very well and much more quickly then AutoClass. Our results indicate that although DBSCAN has lower accuracy compared to K-Means and AutoClass, DBSCAN produces better clusters.
---
paper_title: ACAS: automated construction of application signatures

paper_content:

An accurate mapping of traffic to applications is important for a broad range of network management and measurement tasks. Internet applications have traditionally been identified using well-known default server network-port numbers in the TCP or UDP headers. However this approach has become increasingly inaccurate. An alternate, more accurate technique is to use specific application-level features in the protocol exchange to guide the identification. Unfortunately deriving the signatures manually is very time consuming and difficult.In this paper, we explore automatically extracting application signatures from IP traffic payload content. In particular we apply three statistical machine learning algorithms to automatically identify signatures for a range of applications. The results indicate that this approach is highly accurate and scales to allow online application identification on high speed links. We also discovered that content signatures still work in the presence of encryption. In these cases we were able to derive content signature for unencrypted handshakes negotiating the encryption parameters of a particular connection.
---
paper_title: Unexpected means of protocol inference

paper_content:

Network managers are inevitably called upon to associate network traffic with particular applications. Indeed, this operation is critical for a wide range of management functions ranging from debugging and security to analytics and policy support. Traditionally, managers have relied on application adherence to a well established global port mapping: Web traffic on port 80, mail traffic on port 25 and so on. However, a range of factors - including firewall port blocking, tunneling, dynamic port allocation, and a bloom of new distributed applications - has weakened the value of this approach. We analyze three alternative mechanisms using statistical and structural content models for automatically identifying traffic that uses the same application-layer protocol, relying solely on flow content. In this manner, known applications may be identified regardless of port number, while traffic from one unknown application will be identified as distinct from another. We evaluate each mechanism's classification performance using real-world traffic traces from multiple sites.
---
paper_title: Revealing skype traffic: when randomness plays with you

paper_content:

Skype is a very popular VoIP software which has recently attracted the attention of the research community and network operators. Following a closed source and proprietary design, Skype protocols and algorithms are unknown. Moreover, strong encryption mechanisms are adopted by Skype, making it very difficult to even glimpse its presence from a traffic aggregate. In this paper, we propose a framework based on two complementary techniques to reveal Skypetraffic in real time. The first approach, based on Pearson'sChi-Square test and agnostic to VoIP-related trafficcharacteristics, is used to detect Skype's fingerprint from the packet framing structure, exploiting the randomness introduced at the bit level by the encryption process. Conversely, the second approach is based on a stochastic characterization of Skype traffic in terms of packet arrival rate and packet length, which are used as features of a decision process based on Naive Bayesian Classifiers.In order to assess the effectiveness of the above techniques, we develop an off-line cross-checking heuristic based on deep-packet inspection and flow correlation, which is interesting per se. This heuristic allows us to quantify the amount of false negatives and false positives gathered by means of the two proposed approaches: results obtained from measurements in different networks show that the technique is very effective in identifying Skype traffic. While both Bayesian classifier and packet inspection techniques are commonly used, the idea of leveraging on randomness to reveal traffic is novel. We adopt this to identify Skype traffic, but the same methodology can be applied to other classification problems as well.
---
paper_title: BLINC: multilevel traffic classification in the dark

paper_content:

We present a fundamentally different approach to classifying traffic flows according to the applications that generate them. In contrast to previous methods, our approach is based on observing and identifying patterns of host behavior at the transport layer. We analyze these patterns at three levels of increasing detail (i) the social, (ii) the functional and (iii) the application level. This multilevel approach of looking at traffic flow is probably the most important contribution of this paper. Furthermore, our approach has two important features. First, it operates in the dark, having (a) no access to packet payload, (b) no knowledge of port numbers and (c) no additional information other than what current flow collectors provide. These restrictions respect privacy, technological and practical constraints. Second, it can be tuned to balance the accuracy of the classification versus the number of successfully classified traffic flows. We demonstrate the effectiveness of our approach on three real traces. Our results show that we are able to classify 80%-90% of the traffic with more than 95% accuracy.
---
paper_title: Traffic classification on the fly

paper_content:

The early detection of applications associated with TCP flows is an essential step for network security and traffic engineering. The classic way to identify flows, i.e. looking at port numbers, is not effective anymore. On the other hand, state-of-the-art techniques cannot determine the application before the end of the TCP flow. In this editorial, we propose a technique that relies on the observation of the first five packets of a TCP connection to identify the application. This result opens a range of new possibilities for online traffic classification.
---
paper_title: A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification

paper_content:

The identification of network applications through observation of associated packet traffic flows is vital to the areas of network management and surveillance. Currently popular methods such as port number and payload-based identification exhibit a number of shortfalls. An alternative is to use machine learning (ML) techniques and identify network applications based on per-flow statistics, derived from payload-independent features such as packet length and inter-arrival time distributions. The performance impact of feature set reduction, using Consistency-based and Correlation-based feature selection, is demonstrated on Naive Bayes, C4.5, Bayesian Network and Naive Bayes Tree algorithms. We then show that it is useful to differentiate algorithms based on computational performance rather than classification accuracy alone, as although classification accuracy between the algorithms is similar, computational performance can differ significantly.
---
paper_title: Internet traffic classification using bayesian analysis techniques

paper_content:

Accurate traffic classification is of fundamental importance to numerous other network activities, from security monitoring to accounting, and from Quality of Service to providing operators with useful forecasts for long-term provisioning. We apply a Naive Bayes estimator to categorize traffic by application. Uniquely, our work capitalizes on hand-classified network data, using it as input to a supervised Naive Bayes estimator. In this paper we illustrate the high level of accuracy achievable with the \Naive Bayes estimator. We further illustrate the improved accuracy of refined variants of this estimator.Our results indicate that with the simplest of Naive Bayes estimator we are able to achieve about 65% accuracy on per-flow classification and with two powerful refinements we can improve this value to better than 95%; this is a vast improvement over traditional techniques that achieve 50--70%. While our technique uses training data, with categories derived from packet-content, all of our training and testing was done using header-derived discriminators. We emphasize this as a powerful aspect of our approach: using samples of well-known traffic to allow the categorization of traffic using commonly available information alone.
---
paper_title: Bayesian Neural Networks for Internet Traffic Classification

paper_content:

Internet traffic identification is an important tool for network management. It allows operators to better predict future traffic matrices and demands, security personnel to detect anomalous behavior, and researchers to develop more realistic traffic models. We present here a traffic classifier that can achieve a high accuracy across a range of application types without any source or destination host-address or port information. We use supervised machine learning based on a Bayesian trained neural network. Though our technique uses training data with categories derived from packet content, training and testing were done using features derived from packet streams consisting of one or more packet headers. By providing classification without access to the contents of packets, our technique offers wider application than methods that require full packet/payloads for classification. This is a powerful advantage, using samples of classified traffic to permit the categorization of traffic based only upon commonly available information
---
paper_title: ACAS: automated construction of application signatures

paper_content:

An accurate mapping of traffic to applications is important for a broad range of network management and measurement tasks. Internet applications have traditionally been identified using well-known default server network-port numbers in the TCP or UDP headers. However this approach has become increasingly inaccurate. An alternate, more accurate technique is to use specific application-level features in the protocol exchange to guide the identification. Unfortunately deriving the signatures manually is very time consuming and difficult.In this paper, we explore automatically extracting application signatures from IP traffic payload content. In particular we apply three statistical machine learning algorithms to automatically identify signatures for a range of applications. The results indicate that this approach is highly accurate and scales to allow online application identification on high speed links. We also discovered that content signatures still work in the presence of encryption. In these cases we were able to derive content signature for unencrypted handshakes negotiating the encryption parameters of a particular connection.
---
paper_title: Training on multiple sub-flows to optimise the use of Machine Learning classifiers in real-world IP networks

paper_content:

Literature on the use of machine learning (ML) algorithms for classifying IP traffic has relied on full-flows or the first few packets of flows. In contrast, many real-world scenarios require a classification decision well before a flow has finished even if the flow's beginning is lost. This implies classification must be achieved using statistics derived from the most recent N packets taken at any arbitrary point in a flow's lifetime. We propose training the classifier on a combination of short sub-flows (extracted from full-flow examples of the target application's traffic). We demonstrate this optimisation using the Naive Bayes ML algorithm, and show that our approach results in excellent performance even when classification is initiated mid-way through a flow with windows as small as 25 packets long. We suggest future use of unsupervised ML algorithms to identify optimal sub-flows for training.
---

Title: A Survey of Techniques for Internet Traffic Classification using Machine Learning

Section 1: INTRODUCTION
Description 1: This section introduces the motivation for real-time traffic classification, traditional classification methods and their limitations, and the interest in Machine Learning for traffic classification.

Section 2: The importance of IP traffic classification
Description 2: This section highlights the relevance of IP traffic classification for network management, quality of service schemes, and lawful interception.

Section 3: Traffic classification metrics
Description 3: This section discusses the key metrics used for evaluating traffic classification accuracy, including false positives, false negatives, recall, and precision.

Section 4: Limitations of packet inspection for traffic classification
Description 4: This section outlines the limitations of traditional port-based and payload-based traffic classification methods.

Section 5: Classification based on statistical traffic properties
Description 5: This section introduces the newer approaches to traffic classification that rely on the statistical properties of the traffic.

Section 6: BACKGROUND ON MACHINE LEARNING AND THE APPLICATION OF MACHINE LEARNING IN IP TRAFFIC CLASSIFICATION
Description 6: This section summarizes the basic concepts of Machine Learning, different types of learning, and how they can be applied to IP traffic classification.

Section 7: Challenges for operational deployment
Description 7: This section discusses the challenges and requirements for deploying ML-based classifiers in operational networks, including timely and continuous classification, directional neutrality, efficient use of memory and processors, and portability and robustness.

Section 8: A REVIEW OF MACHINE LEARNING BASED IP TRAFFIC CLASSIFICATION TECHNIQUES
Description 8: This section reviews significant works on ML-based IP traffic classification, categorized into clustering approaches, supervised learning approaches, and hybrid approaches.

Section 9: Comparisons and Related Work
Description 9: This section compares the performance and efficacy of different ML algorithms and approaches for traffic classification.

Section 10: Qualitative summary
Description 10: This section provides a qualitative summary of the reviewed works against critical operational requirements for real-time classifiers and discusses open issues and future research directions.