Big Data Analytics for Wireless and Wired Network Design: A Survey
---
paper_title: A survey on indexing techniques for big data: taxonomy and performance evaluation

paper_content:

The explosive growth in volume, velocity, and diversity of data produced by mobile devices and cloud applications has contributed to the abundance of data or `big data.' Available solutions for efficient data storage and management cannot fulfill the needs of such heterogeneous data where the amount of data is continuously increasing. For efficient retrieval and management, existing indexing solutions become inefficient with the rapidly growing index size and seek time and an optimized index scheme is required for big data. Regarding real-world applications, the indexing issue with big data in cloud computing is widespread in healthcare, enterprises, scientific experiments, and social networks. To date, diverse soft computing, machine learning, and other techniques in terms of artificial intelligence have been utilized to satisfy the indexing requirements, yet in the literature, there is no reported state-of-the-art survey investigating the performance and consequences of techniques for solving indexing in big data issues as they enter cloud computing. The objective of this paper is to investigate and examine the existing indexing techniques for big data. Taxonomy of indexing techniques is developed to provide insight to enable researchers understand and select a technique as a basis to design an indexing mechanism with reduced time and space consumption for BD-MCC. In this study, 48 indexing techniques have been studied and compared based on 60 articles related to the topic. The indexing techniques' performance is analyzed based on their characteristics and big data indexing requirements. The main contribution of this study is taxonomy of categorized indexing techniques based on their method. The categories are non-artificial intelligence, artificial intelligence, and collaborative artificial intelligence indexing methods. In addition, the significance of different procedures and performance is analyzed, besides limitations of each technique. In conclusion, several key future research topics with potential to accelerate the progress and deployment of artificial intelligence-based cooperative indexing in BD-MCC are elaborated on.
---
paper_title: Traffic Flow Prediction With Big Data: A Deep Learning Approach

paper_content:

Accurate and timely traffic flow information is important for the successful deployment of intelligent transportation systems. Over the last few years, traffic data have been exploding, and we have truly entered the era of big data for transportation. Existing traffic flow prediction methods mainly use shallow traffic prediction models and are still unsatisfying for many real-world applications. This situation inspires us to rethink the traffic flow prediction problem based on deep architecture models with big traffic data. In this paper, a novel deep-learning-based traffic flow prediction method is proposed, which considers the spatial and temporal correlations inherently. A stacked autoencoder model is used to learn generic traffic flow features, and it is trained in a greedy layerwise fashion. To the best of our knowledge, this is the first time that a deep architecture model is applied using autoencoders as building blocks to represent traffic flow features for prediction. Moreover, experiments demonstrate that the proposed method for traffic flow prediction has superior performance.
---
paper_title: Big Data: Issues and Challenges Moving Forward

paper_content:

Big data refers to data volumes in the range of exabytes (1018) and beyond. Such volumes exceed the capacity of current on-line storage systems and processing systems. Data, information, and knowledge are being created and collected at a rate that is rapidly approaching the exabyte/year range. But, its creation and aggregation are accelerating and will approach the zettabyte/year range within a few years. Volume is only one aspect of big data; other attributes are variety, velocity, value, and complexity. Storage and data transport are technology issues, which seem to be solvable in the near-term, but represent longterm challenges that require research and new paradigms. We analyze the issues and challenges as we begin a collaborative research program into methodologies for big data analysis and design.
---
paper_title: Big data, as part of the critical infrastructure

paper_content:

The amount of information and the need for information is fast increasing today, and it will further multiply in the coming years. The critical infrastructure ensures the welfare of the state and its citizens. Everyday decision-making requires a large amount of information of various types. The present publication will show the relationship between the critical infrastructure and Big Data. The emergence and use of Big Data have been greatly emphasized in recent years. The present publication will present the conceptual and practical manifestations of the Big Data phenomenon. The history of Big Data will also be discussed, as well as the conditions and possibilities of its applications, which will serve for the next period of economic growth. It will also present the Digital National Development Program, funded by the Government of Hungary and the European Union to increase the welfare of the Hungarian society and to make Big Data a digital service that is easier to access for everyone. Furthermore, it will examine the security and risk factors of Big Data, which is essential to ensure the effective protection of the critical infrastructure and personal data.
---
paper_title: The essentials of computer organization and architecture

paper_content:

Updated and revised to reflect the most current data in the field, perennial bestseller The Essentials of Computer Organization and Architecture, Fourth Edition is comprehensive enough to address all necessary organization and architecture topics, but concise enough to be appropriate for a single-term course. Its focus on real-world examples and practical applications encourages students to develop a big-picture understanding of how essential organization and architecture concepts are applied in the computing world. In addition to direct correlation with the ACM/IEEE CS2013 guidelines for computer organization and architecture, the text exposes readers to the inner workings of a modern digital computer through an integrated presentation of fundamental concepts and principles. The fully revised and updated Fourth Edition includes the most up-to-the-minute data and resources available and reflects current technologies, including tablets and cloud computing. All-new exercises, expanded discussions, and feature boxes in every chapter implement even more real-world applications and current data, and many chapters include all-new examples. A full suite of student and instructor resources, including a secure companion website, Lecture Outlines in PowerPoint Format, and an Instructor Manual, complement the text. This award-winning, best-selling text is the most thorough, student-friendly, and accessible text on the market today. Key Features: The Fourth Edition is in direct correlation with the ACM/IEEE CS2013 guidelines for computer organization and architecture, in addition to integrating material from additional knowledge units. All-new material on a variety of topics, including zetabytes and yottabytes, automatons, tablet computers, graphic processing units, and cloud computing The MARIE Simulator package allows students to learn the essential concepts of computer organization and architecture, including assembly language, without getting caught up in unnecessary and confusing details. Full suite of ancillary materials, including a secure companion website, PowerPoint lecture outlines, and an Instructor Manual Bundled with an optional Intel supplement Ideally suited for single-term courses
---
paper_title: OverFlow: Multi-Site Aware Big Data Management for Scientific Workflows on Clouds

paper_content:

The global deployment of cloud datacenters is enabling large scale scientific workflows to improve performance and deliver fast responses. This unprecedented geographical distribution of the computation is doubled by an increase in the scale of the data handled by such applications, bringing new challenges related to the efficient data management across sites. High throughput, low latencies or cost-related trade-offs are just a few concerns for both cloud providers and users when it comes to handling data across datacenters. Existing solutions are limited to cloud-provided storage, which offers low performance based on rigid cost schemes. In turn, workflow engines need to improvise substitutes, achieving performance at the cost of complex system configurations, maintenance overheads, reduced reliability and reusability. In this paper, we introduce OverFlow, a uniform data management system for scientific workflows running across geographically distributed sites, aiming to reap economic benefits from this geo-diversity. Our solution is environment-aware, as it monitors and models the global cloud infrastructure, offering high and predictable data handling performance for transfer cost and time, within and across sites. OverFlow proposes a set of pluggable services, grouped in a data scientist cloud kit. They provide the applications with the possibility to monitor the underlying infrastructure, to exploit smart data compression, deduplication and geo-replication, to evaluate data management costs, to set a tradeoff between money and time, and optimize the transfer strategy accordingly. The system was validated on the Microsoft Azure cloud across its 6 EU and US datacenters. The experiments were conducted on hundreds of nodes using synthetic benchmarks and real-life bio-informatics applications (A-Brain, BLAST). The results show that our system is able to model accurately the cloud performance and to leverage this for efficient data dissemination, being able to reduce the monetary costs and transfer time by up to three times.
---
paper_title: Sustainable Development Plan for Korea through Expansion of Green IT: Policy Issues for the Effective Utilization of Big Data

paper_content:

The South Korean government is providing full support for green IT as one of the growth engines of Korea. The purpose of this study is to derive policy issues needed for the sustainable development of Korea through utilizing Big Data by applying green IT. The analysis is done using a Delphi technique. Results show that the establishment of computing platforms that can easily share data and generate value is prioritized for the effective use of Big Data from the environment. In addition, the government-led publication of genetic information and electronic medical records for research purposes has been derived as an important policy issue for the use of bio-Big Data. Besides, a guideline concerning the standardization of machine to machine and Internet of Things communication and data security is needed to effectively use Big Data from machines/things. Moreover, a review of legislation related to the utilization of Big Data from digital media has been derived as an important policy issue. The results of this study propose the direction in which the Korean government should move for green growth through effective utilization of Big Data. The results can be also useful resources for establishing relevant policies for various countries that are accelerating sustainable development.
---
paper_title: The Google file system

paper_content:

We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.
---
paper_title: Big data for IoT services in smart cities

paper_content:

This paper analyzes the benefits of big data for smart cities and the potential of the knowledge discovery from sensed data. Big data enables real-time systems monitoring, management, optimization and anticipation. In this work we present some examples of applications of big data analysis in two scenarios of smart cities. One of them describes the services provided in the SmartCampus of the University of Murcia. The second example is focused on a tram service scenario where thousands of transit-card transactions should be processed. The results obtained after applying the most appropriate big data techniques in both scenarios show how it is possible to provide efficiently services like the management of the energy consumption and comfort in buildings, and the transport congestion in the context of smart cities.
---
paper_title: A survey of open source tools for machine learning with big data in the Hadoop ecosystem

paper_content:

With an ever-increasing amount of options, the task of selecting machine learning tools for big data can be difficult. The available tools have advantages and drawbacks, and many have overlapping uses. The world’s data is growing rapidly, and traditional tools for machine learning are becoming insufficient as we move towards distributed and real-time processing. This paper is intended to aid the researcher or professional who understands machine learning but is inexperienced with big data. In order to evaluate tools, one should have a thorough understanding of what to look for. To that end, this paper provides a list of criteria for making selections along with an analysis of the advantages and drawbacks of each. We do this by starting from the beginning, and looking at what exactly the term “big data” means. From there, we go on to the Hadoop ecosystem for a look at many of the projects that are part of a typical machine learning architecture and an understanding of how everything might fit together. We discuss the advantages and disadvantages of three different processing paradigms along with a comparison of engines that implement them, including MapReduce, Spark, Flink, Storm, and H2O. We then look at machine learning libraries and frameworks including Mahout, MLlib, SAMOA, and evaluate them based on criteria such as scalability, ease of use, and extensibility. There is no single toolkit that truly embodies a one-size-fits-all solution, so this paper aims to help make decisions smoother by providing as much information as possible and quantifying what the tradeoffs will be. Additionally, throughout this paper, we review recent research in the field using these tools and talk about possible future directions for toolkit-based learning.
---
paper_title: A survey on indexing techniques for big data: taxonomy and performance evaluation

paper_content:

The explosive growth in volume, velocity, and diversity of data produced by mobile devices and cloud applications has contributed to the abundance of data or `big data.' Available solutions for efficient data storage and management cannot fulfill the needs of such heterogeneous data where the amount of data is continuously increasing. For efficient retrieval and management, existing indexing solutions become inefficient with the rapidly growing index size and seek time and an optimized index scheme is required for big data. Regarding real-world applications, the indexing issue with big data in cloud computing is widespread in healthcare, enterprises, scientific experiments, and social networks. To date, diverse soft computing, machine learning, and other techniques in terms of artificial intelligence have been utilized to satisfy the indexing requirements, yet in the literature, there is no reported state-of-the-art survey investigating the performance and consequences of techniques for solving indexing in big data issues as they enter cloud computing. The objective of this paper is to investigate and examine the existing indexing techniques for big data. Taxonomy of indexing techniques is developed to provide insight to enable researchers understand and select a technique as a basis to design an indexing mechanism with reduced time and space consumption for BD-MCC. In this study, 48 indexing techniques have been studied and compared based on 60 articles related to the topic. The indexing techniques' performance is analyzed based on their characteristics and big data indexing requirements. The main contribution of this study is taxonomy of categorized indexing techniques based on their method. The categories are non-artificial intelligence, artificial intelligence, and collaborative artificial intelligence indexing methods. In addition, the significance of different procedures and performance is analyzed, besides limitations of each technique. In conclusion, several key future research topics with potential to accelerate the progress and deployment of artificial intelligence-based cooperative indexing in BD-MCC are elaborated on.
---
paper_title: Addressing big data issues in Scientific Data Infrastructure

paper_content:

Big Data are becoming a new technology focus both in science and in industry. This paper discusses the challenges that are imposed by Big Data on the modern and future Scientific Data Infrastructure (SDI). The paper discusses a nature and definition of Big Data that include such features as Volume, Velocity, Variety, Value and Veracity. The paper refers to different scientific communities to define requirements on data management, access control and security. The paper introduces the Scientific Data Lifecycle Management (SDLM) model that includes all the major stages and reflects specifics in data management in modern e-Science. The paper proposes the SDI generic architecture model that provides a basis for building interoperable data or project centric SDI using modern technologies and best practices. The paper explains how the proposed models SDLM and SDI can be naturally implemented using modern cloud based infrastructure services provisioning model and suggests the major infrastructure components for Big Data.
---
paper_title: Big Data: Issues and Challenges Moving Forward

paper_content:

Big data refers to data volumes in the range of exabytes (1018) and beyond. Such volumes exceed the capacity of current on-line storage systems and processing systems. Data, information, and knowledge are being created and collected at a rate that is rapidly approaching the exabyte/year range. But, its creation and aggregation are accelerating and will approach the zettabyte/year range within a few years. Volume is only one aspect of big data; other attributes are variety, velocity, value, and complexity. Storage and data transport are technology issues, which seem to be solvable in the near-term, but represent longterm challenges that require research and new paradigms. We analyze the issues and challenges as we begin a collaborative research program into methodologies for big data analysis and design.
---
paper_title: Virtual network topology adaptability based on data analytics for traffic prediction

paper_content:

The introduction of new services requiring large and dynamic bitrate connectivity can cause changes in the direction of the traffic in metro and even core network segments throughout the day. This leads to large overprovisioning in statically managed virtual network topologies (VNTs), which are designed to cope with the traffic forecast. To reduce expenses while ensuring the required grade of service, in this paper we propose a VNT reconfiguration approach based on data analytics for traffic prediction (VENTURE). It regularly reconfigures the VNT based on the predicted traffic, thus adapting the topology to both the current and the predicted traffic volume and direction. A machine learning algorithm based on an artificial neural network is used to provide robust and adaptive traffic models. The reconfiguration problem that takes as its input the traffic prediction is modeled mathematically, and a heuristic is proposed to solve it in practical times. To support VENTURE, we propose an architecture that allows collecting and storing data from monitoring at the routers and that is used to train predictive models for every origin-destination pair. Exhaustive simulation results of the algorithm, together with the experimental assessment of the proposed architecture, are finally presented.
---
paper_title: Managing cloud-generated logs using big data technologies

paper_content:

Cloud computing is a fast-growing paradigm that has forcefully emerged and established itself in the next generation of IT industry and business, performing massive-scale and complex computing. As cloud technology adoption continues to progress, massive growth in the scale of data generated through cloud computing has been observed. A large part of that data has been machine-generated log data; it is generated at every layer and component within the cloud information technology ecosystem that spans a range of IT functions from storage and computation to networking and application services. Log files are essential for enterprise-level monitoring, troubleshooting, security, debugging, compliance, etc. We argue that cloud computing is in need of a new methodology for recording, managing, storing and analyzing the log data that is generated by the cloud infrastructure, based on big data technologies, allowing for the access to a wealth of knowledge that can contribute to its own advancement. Our paper discusses and expands existing standard log management solutions to design new methods of exploiting the unstructured log data in order to gain unprecedented insights.
---
paper_title: Toward Scalable Systems for Big Data Analytics: A Technology Tutorial

paper_content:

Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.
---
paper_title: Hadoop: The Definitive Guide

paper_content:

Hadoop: The Definitive Guide helps you harness the power of your data. Ideal for processing large datasets, the Apache Hadoop framework is an open source implementation of the MapReduce algorithm on which Google built its empire. This comprehensive resource demonstrates how to use Hadoop to build reliable, scalable, distributed systems: programmers will find details for analyzing large datasets, and administrators will learn how to set up and run Hadoop clusters. Complete with case studies that illustrate how Hadoop solves specific problems, this book helps you: Use the Hadoop Distributed File System (HDFS) for storing large datasets, and run distributed computations over those datasets using MapReduce Become familiar with Hadoop's data and I/O building blocks for compression, data integrity, serialization, and persistence Discover common pitfalls and advanced features for writing real-world MapReduce programs Design, build, and administer a dedicated Hadoop cluster, or run Hadoop in the cloud Use Pig, a high-level query language for large-scale data processing Take advantage of HBase, Hadoop's database for structured and semi-structured data Learn ZooKeeper, a toolkit of coordination primitives for building distributed systems If you have lots of data -- whether it's gigabytes or petabytes -- Hadoop is the perfect solution. Hadoop: The Definitive Guide is the most thorough book available on the subject. "Now you have the opportunity to learn about Hadoop from a master-not only of the technology, but also of common sense and plain talk." -- Doug Cutting, Hadoop Founder, Yahoo!
---
paper_title: Towards a cost-efficient MapReduce: Mitigating power peaks for Hadoop clusters

paper_content:

Distributed data processing system is becoming one of the most important components for data-intensive computational tasks in the enterprise software infrastructure. Deploying and operating such systems require large amount of costs, including hardware costs to build clusters and energy costs to run clusters. To make these systems sustainable and scalable, power management has been an important research problem. In this paper, we take Hadoop as an example to illustrate the power peak problem which causes power inefficiency and provides in-depth analysis to explain issues with existing system designs. We propose a novel power capping module in the Hadoop scheduler to mitigate power peaks. Extensive simulation studies show that our proposed solution can effectively smooth the power consumption curve and mitigate temporary power peaks for Hadoop clusters.
---
paper_title: Big Data Analytics: Methods and Applications

paper_content:

This book has a collection of articles written by Big Data experts to describe some of the cutting-edge methods and applications from their respective areas of interest, and provides the reader with adetailed overview of the field of Big Data Analytics as it is practiced today. The chapterscover technical aspects of key areasthat generate and use Big Data such as management andfinance; medicine andhealthcare; genome, cytome and microbiome;graphs and networks; Internet of Things; Big Data standards; bench-marking of systems; and others. In addition to different applications, key algorithmic approaches such as graph partitioning, clustering and finite mixture modelling ofhigh-dimensional data are also covered. The varied collection of themes in this volume introduces the reader to the richnessof the emerging field of Big Data Analytics.
---
paper_title: ARLS: A MapReduce-based output analysis tool for large-scale simulations

paper_content:

Abstract As simulations are becoming popular in the analysis of the complex behavior of large-scale systems with immense inputs and outputs, there is an increasing demand to efficiently store, manage, and analyze massive simulation outputs. Hadoop and MapReduce have been used in various applications to speed up the process of analyzing large amounts of datasets. In this paper, we present ARLS (After-action Reviewer for Large-scale Simulations), a MapReduce-based output analysis tool for simulation outputs. ARLS clusters distributed storages using Hadoop and automatically composes Map and Reduce functions to process the simulation outputs. ARLS has been applied to our SAM (Surface-to-Air Missile) simulator. The SAM simulator has been developed to analyze the dynamics of a missile in designing air-defense systems. ARLS takes a large amount of unstructured simulation outputs from SAM simulator, automatically generates Map and Reduce functions to analyze the missile and the aircraft component of SAM simulator, and executes Map and Reduce jobs in parallel. The results of our experiments show that ARLS can efficiently analyze a large amount of unstructured simulation datasets by distributing datasets and computations over the cluster of commodity machines.
---
paper_title: MapReduce Based Text Detection in Big Data Natural Scene Videos

paper_content:

Abstract Text detection is a basic step in many computer vision applications including video Optical Character Recognition (OCR), video indexation, understanding video content, etc. Actually, several sources such as mobile devices, monitoring cameras and social networks are generating every day billions of videos with different formats and uncertainty. Such videos require new methods to apply text detection. In this paper, we are introducing a novel text detection technique using many blocks coming from frame decomposition. Each block is analyzed and classified which allowed the extraction of text coordinates using MapReduce programming model. To validate our approach, we test it on YouTube Video Text (YVT) dataset and we found that the running speed of this approach can be more than 2 times as fast as classic approach.
---
paper_title: Big Data for Health

paper_content:

This paper provides an overview of recent developments in big data in the context of biomedical and health informatics. It outlines the key characteristics of big data and how medical and health informatics, translational bioinformatics, sensor informatics, and imaging informatics will benefit from an integrated approach of piecing together different aspects of personalized information from a diverse range of data sources, both structured and unstructured, covering genomics, proteomics, metabolomics, as well as imaging, clinical diagnosis, and long-term continuous physiological sensing of an individual. It is expected that recent advances in big data will expand our knowledge for testing new hypotheses about disease management from diagnosis to prevention to personalized treatment. The rise of big data, however, also raises challenges in terms of privacy, security, data ownership, data stewardship, and governance. This paper discusses some of the existing activities and future opportunities related to big data for health, outlining some of the key underlying issues that need to be tackled.
---
paper_title: Big Data Open Source Platforms

paper_content:

In a global market the capacity to mine and analyze user data is one way for companies to be as close in time and accuracy to the needs of their users. Big Data Platforms are one solution for companies to solve the necessary challenges to accomplish these capacities. Unfortunately the number of challenges that need to be addressed, allied with the high number of different solutions proposed, has led to the creation of a high number of different platforms making it hard to name one definitive and adequate platform for companies. In this paper we compare six of the most important Big Data Open Source Platforms to help companies or organizations choose the most adequate one to their needs. We analyze the following open source platforms - Apache Mahout, MOA, R Project, Vow pal Wabbit, PEGASUS and Graph Lab Create TM.
---
paper_title: A survey of open source tools for machine learning with big data in the Hadoop ecosystem

paper_content:

With an ever-increasing amount of options, the task of selecting machine learning tools for big data can be difficult. The available tools have advantages and drawbacks, and many have overlapping uses. The world’s data is growing rapidly, and traditional tools for machine learning are becoming insufficient as we move towards distributed and real-time processing. This paper is intended to aid the researcher or professional who understands machine learning but is inexperienced with big data. In order to evaluate tools, one should have a thorough understanding of what to look for. To that end, this paper provides a list of criteria for making selections along with an analysis of the advantages and drawbacks of each. We do this by starting from the beginning, and looking at what exactly the term “big data” means. From there, we go on to the Hadoop ecosystem for a look at many of the projects that are part of a typical machine learning architecture and an understanding of how everything might fit together. We discuss the advantages and disadvantages of three different processing paradigms along with a comparison of engines that implement them, including MapReduce, Spark, Flink, Storm, and H2O. We then look at machine learning libraries and frameworks including Mahout, MLlib, SAMOA, and evaluate them based on criteria such as scalability, ease of use, and extensibility. There is no single toolkit that truly embodies a one-size-fits-all solution, so this paper aims to help make decisions smoother by providing as much information as possible and quantifying what the tradeoffs will be. Additionally, throughout this paper, we review recent research in the field using these tools and talk about possible future directions for toolkit-based learning.
---
paper_title: Challenges in 5G: how to empower SON with big data for enabling 5G

paper_content:

While an al dente character of 5G is yet to emerge, network densification, miscellany of node types, split of control and data plane, network virtualization, heavy and localized cache, infrastructure sharing, concurrent operation at multiple frequency bands, simultaneous use of different medium access control and physical layers, and flexible spectrum allocations can be envisioned as some of the potential ingredients of 5G. It is not difficult to prognosticate that with such a conglomeration of technologies, the complexity of operation and OPEX can become the biggest challenge in 5G. To cope with similar challenges in the context of 3G and 4G networks, recently, self-organizing networks, or SONs, have been researched extensively. However, the ambitious quality of experience requirements and emerging multifarious vision of 5G, and the associated scale of complexity and cost, demand a significantly different, if not totally new, approach toward SONs in order to make 5G technically as well as financially feasible. In this article we first identify what challenges hinder the current self-optimizing networking paradigm from meeting the requirements of 5G. We then propose a comprehensive framework for empowering SONs with big data to address the requirements of 5G. Under this framework we first characterize big data in the context of future mobile networks, identifying its sources and future utilities. We then explicate the specific machine learning and data analytics tools that can be exploited to transform big data into the right data that provides a readily useable knowledge base to create end-to-end intelligence of the network. We then explain how a SON engine can build on the dynamic models extractable from the right data. The resultant dynamicity of a big data empowered SON (BSON) makes it more agile and can essentially transform the SON from being a reactive to proactive paradigm and hence act as a key enabler for 5G's extremely low latency requirements. Finally, we demonstrate the key concepts of our proposed BSON framework through a case study of a problem that the classic 3G/4G SON fails to solve.
---
paper_title: Big data analytics for network anomaly detection from netflow data

paper_content:

Cyber-attacks wasorganized in a simple and random way in the past. However attacks are carried out systematically and long term nowadays. In addition, the high calculation volume and continuous changes in network data distribution have made it more difficult to analyze data and detect abnormal behaviors within. For this reason, big data solutions have become essential. In this paper, firstly network anomaly and attack detection studies on big data has been reviewed. Then, a public big network data was analyzed with a new unsupervised anomaly detection approach on Apache Spark cluster in Azure HDInsight. Finally, the results obtained from a case study were evaluated, %96 accuracy was achieved. The results were visualized after dimension reduction using Principal Component Analysis (PCA). The identified anomalies may provide usable outputs to understand the behavior of the network, distinguishing the attacks, providing better cyber security, and protecting critical infrastructures.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: Anomaly Detection In Cellular Network Data Using Big Data Analytics

paper_content:

Anomaly detection is a key component in which perturbations from a normal behavior suggests a misconfigured/ mismatched data in related systems. In this paper, we present a call detail record based anomaly detection method (CADM) that analyzes the users's calling activities and detects the abnormal behavior of user movements in a real cellular network. CADM is capable of detecting the location of the site that an anomaly has occurred. We evaluate the proposed CADM by performing experiments over the call-detail records of AVEA, a mobile service provider in Turkey.
---
paper_title: LTE performance data reduction for knowledge acquisition

paper_content:

In the last years, mobile networks have seen a great increase in complexity, as the data traffic, the demand for quality and the variety of offered services have grown. The management costs of modern networks are growing, at the same time as operators compete to offer shorter downtime and less impact of network issues on the user experience. Self-Organizing Networks (SON) offer a solution to these problems. Among these SON functionalities in cellular networks, self-healing automates the resolution of problems in the radio access network. To perform the task of diagnosis (or root cause analysis), Knowledge-Based Systems (KBS) are often used. These systems need a previous process of training (or learning), in which they are fed instances of real problems. In this paper, an algorithm for extracting the key information for these vectors is proposed. The inputs of the algorithm are big matrices of time-dependent network performance data, and the outputs are simple one-dimensional vectors ready to be used in learning algorithms.
---
paper_title: Monitoring and analyzing big traffic data of a large-scale cellular network with Hadoop

paper_content:

Network traffic monitoring and analysis is of theoretical and practical significance for optimizing network resource and improving user experience. However, existing solutions, which usually rely on a high-performance server with large storage capacity, are not scalable for detailed analysis of a large volume of traffic data. In this article, we present a traffic monitoring and analysis system for large-scale networks based on Hadoop, an open-source distributed computing platform for big data processing on commodity hardware. This system has been deployed in the core network of a large cellular network and extensively evaluated. The results demonstrate that the system can efficiently processes 4.2 Tbytes of traffic data from 123 Gb/s links with high performance and low cost every day.
---
paper_title: Mobile Internet big data platform in China Unicom

paper_content:

China Unicom, the largest WCDMA 3G operator in China, meets the requirements of the historical Mobile Internet Explosion, or the surging of Mobile Internet Traffic from mobile terminals. According to the internal statistics of China Unicom, mobile user traffic has increased rapidly with a Compound Annual Growth Rate (CAGR) of 135%. Currently China Unicom monthly stores more than 2 trillion records, data volume is over 525 TB, and the highest data volume has reached a peak of 5 PB. Since October 2009, China Unicom has been developing a home-brewed big data storage and analysis platform based on the open source Hadoop Distributed File System (HDFS) as it has a long-term strategy to make full use of this Big Data. All Mobile Internet Traffic is well served using this big data platform. Currently, the writing speed has reached 1 390 000 records per second, and the record retrieval time in the table that contains trillions of records is less than 100 ms. To take advantage of this opportunity to be a Big Data Operator, China Unicom has developed new functions and has multiple innovations to solve space and time constraint challenges presented in data processing. In this paper, we will introduce our big data platform in detail. Based on this big data platform, China Unicom is building an industry ecosystem based on Mobile Internet Big Data, and considers that a telecom operator centric ecosystem can be formed that is critical to reach prosperity in the modern communications business.
---
paper_title: A dynamic bandwidth allocation algorithm in mobile networks with big data of users and networks

paper_content:

Data collection has become easy due to the rapid development of both mobile devices and wireless networks. In each second, numerous data are generated by user devices and collected through wireless networks. These data, carrying user and network related information, are invaluable for network management. However, they were seldom employed to improve network performance in existing research work. In this article we propose a bandwidth allocation algorithm to increase the throughput of cellular network users by exploring user and network data collected from user devices. With the aid of these data, users can be categorized into clusters and share bandwidth to improve the resource utilization of the network. Simulation results indicate that the proposed scheme is able to rationally form clusters among mobile users and thus significantly increase the throughput and bandwidth efficiency of the network.
---
paper_title: Improving Data Extraction Efficiency of Cache Nodes in Cognitive Radio Networks Using Big Data Analysis

paper_content:

In cognitive radio networks, unlicensed users are allowed to use underutilized licensed spectrum until licensed users' transmission quality of service is not compromised. As soon as the conflict goes beyond a certain limit, SU must leave the spectrum and move to the other nearby free band. At the time of interruption, sensing the nearby free channels and switching to them will take some time, hence the ongoing data will be interrupted, which will delay the data transmission. To minimize this delay, creating cache of the SU signal at multiple nodes in a cluster has shown significant improvement in reducing the transmission delay if cache placement is done systematically. This systematic and accurate placement of cache is possible if the data accumulated is accessed and processed quickly. Taking into account the vastness of cluster networks, a huge amount of data will be required to be accessed and processed. Cognitive Radio networks are very complex structures when it comes to the information sharing amongst the secondary users and with the cluster head. Taking into account, whether unlicensed users share their information with other secondary users, and in case if they do, how much proportion of it they allow the fusion center to process, several big data scenarios exist. This paper discusses the possible information sharing scenarios in cognitive radio network systems and their possible Big Data Solutions.
---
paper_title: Big Data Meets Telcos: A Proactive Caching Perspective

paper_content:

Mobile cellular networks are becoming increasingly complex to manage while classical deployment/optimization techniques and current solutions (i.e., cell densification, acquiring more spectrum, etc.) are cost-ineffective and thus seen as stopgaps. This calls for development of novel approaches that leverage recent advances in storage/memory, context-awareness, edge/cloud computing, and falls into framework of big data. However, the big data by itself is yet another complex phenomena to handle and comes with its notorious 4V: velocity, voracity, volume and variety. In this work, we address these issues in optimization of 5G wireless networks via the notion of proactive caching at the base stations. In particular, we investigate the gains of proactive caching in terms of backhaul offloadings and request satisfactions, while tackling the large-amount of available data for content popularity estimation. In order to estimate the content popularity, we first collect users' mobile traffic data from a Turkish telecom operator from several base stations in hours of time interval. Then, an analysis is carried out locally on a big data platform and the gains of proactive caching at the base stations are investigated via numerical simulations. It turns out that several gains are possible depending on the level of available information and storage size. For instance, with 10% of content ratings and 15.4 Gbyte of storage size (87% of total catalog size), proactive caching achieves 100% of request satisfaction and offloads 98% of the backhaul when considering 16 base stations.
---
paper_title: Big data-driven optimization for mobile networks toward 5G

paper_content:

Big data offers a plethora of opportunities to mobile network operators for improving quality of service. This article explores various means of integrating big data analytics with network optimization toward the objective of improving the user quality of experience. We first propose a framework of Big Data-Driven (BDD) mobile network optimization. We then present the characteristics of big data that are collected not only from user equipment but also from mobile networks. Moreover, several techniques in data collection and analytics are discussed from the viewpoint of network optimization. Certain user cases on the application of the proposed framework for improving network performance are also given in order to demonstrate the feasibility of the framework. With the integration of the emerging fifth generation (5G) mobile networks with big data analytics, the quality of our daily mobile life is expected to be tremendously enhanced.
---
paper_title: Big data-driven optimization for mobile networks toward 5G

paper_content:

Big data offers a plethora of opportunities to mobile network operators for improving quality of service. This article explores various means of integrating big data analytics with network optimization toward the objective of improving the user quality of experience. We first propose a framework of Big Data-Driven (BDD) mobile network optimization. We then present the characteristics of big data that are collected not only from user equipment but also from mobile networks. Moreover, several techniques in data collection and analytics are discussed from the viewpoint of network optimization. Certain user cases on the application of the proposed framework for improving network performance are also given in order to demonstrate the feasibility of the framework. With the integration of the emerging fifth generation (5G) mobile networks with big data analytics, the quality of our daily mobile life is expected to be tremendously enhanced.
---
paper_title: Enhanced inter-cell interference coordination in co-channel multi-layer LTE-advanced networks

paper_content:

Different technical solutions and innovations are enabling the move from macro-only scenarios towards heterogeneous networks with a mixture of different base station types. In this article we focus on multi-layer LTE-Advanced networks, and especially address aspects related to interference management. The network controlled time-domain enhanced inter-cell interference coordination (eICIC) concept is outlined by explaining the benefits and characteristics of this solution. The benefits of using advanced terminal device receiver architectures with interference suppression capabilities are motivated. Extensive system level performance results are presented with bursty traffic to demonstrate the eICIC concepts ability to dynamically adapt according to the traffic conditions.
---
paper_title: Big data-driven optimization for mobile networks toward 5G

paper_content:

Big data offers a plethora of opportunities to mobile network operators for improving quality of service. This article explores various means of integrating big data analytics with network optimization toward the objective of improving the user quality of experience. We first propose a framework of Big Data-Driven (BDD) mobile network optimization. We then present the characteristics of big data that are collected not only from user equipment but also from mobile networks. Moreover, several techniques in data collection and analytics are discussed from the viewpoint of network optimization. Certain user cases on the application of the proposed framework for improving network performance are also given in order to demonstrate the feasibility of the framework. With the integration of the emerging fifth generation (5G) mobile networks with big data analytics, the quality of our daily mobile life is expected to be tremendously enhanced.
---
paper_title: Big data-driven optimization for mobile networks toward 5G

paper_content:

Big data offers a plethora of opportunities to mobile network operators for improving quality of service. This article explores various means of integrating big data analytics with network optimization toward the objective of improving the user quality of experience. We first propose a framework of Big Data-Driven (BDD) mobile network optimization. We then present the characteristics of big data that are collected not only from user equipment but also from mobile networks. Moreover, several techniques in data collection and analytics are discussed from the viewpoint of network optimization. Certain user cases on the application of the proposed framework for improving network performance are also given in order to demonstrate the feasibility of the framework. With the integration of the emerging fifth generation (5G) mobile networks with big data analytics, the quality of our daily mobile life is expected to be tremendously enhanced.
---
paper_title: Resource allocation optimization in LTE-A/5G networks using big data analytics

paper_content:

The next generation wireless cellular communication networks are going to be completely internet protocol (IP) based. This leads to evolution of cellular networks into software controlled hierarchical abstract networks. Software controlling enables extraction of large volumes of network usage and transactions related information in the form of log files, configuration files, database entries etc. This paper proposes a mechanism for efficient exploitation of this big volume of data for addressing the ever existing problem of optimum radio resource allocation to users. This work is based on the map-reduce data processing technique, by implementing non-orthogonal binary singular value decomposition (SVD) of binary data matrices for fuzzy pattern recognition and grouping. Patterns identified by proposed method can be interpreted efficiently for resource allocation in wireless networks.
---
paper_title: FUZZY PATTERN MATCHING

paper_content:

Pattern‐directed inference systems (P.D.I.S.) are among the most largely used tools in A.I. to‐day in order to represent and exploit knowledge. Generally, P.D.I.S.'s use production rules triggered by matching between rule patterns and elements of the data base. However, the lack of flexibility in the matching remains a drawback in this kind of system. In the framework of the communication in natural language with robots, approximate descriptions of real world situations and approximately specified rules are needed; furthermore, similarity in the matching process does not always need to be perfect. Thus, the pervading fuzziness of natural language can be taken into account. The following levels, belonging to the real interval [0,1], are evaluated: The possibility of similarity between referents designated in the data and in the pattern respectively; the necessity that a referent designated in the data is similar to a referent designated in the pattern. Designations are fuzzy when the pattern or the data are fuzzy, which is usual with words of a natural language.
---
paper_title: Challenges in 5G: how to empower SON with big data for enabling 5G

paper_content:

While an al dente character of 5G is yet to emerge, network densification, miscellany of node types, split of control and data plane, network virtualization, heavy and localized cache, infrastructure sharing, concurrent operation at multiple frequency bands, simultaneous use of different medium access control and physical layers, and flexible spectrum allocations can be envisioned as some of the potential ingredients of 5G. It is not difficult to prognosticate that with such a conglomeration of technologies, the complexity of operation and OPEX can become the biggest challenge in 5G. To cope with similar challenges in the context of 3G and 4G networks, recently, self-organizing networks, or SONs, have been researched extensively. However, the ambitious quality of experience requirements and emerging multifarious vision of 5G, and the associated scale of complexity and cost, demand a significantly different, if not totally new, approach toward SONs in order to make 5G technically as well as financially feasible. In this article we first identify what challenges hinder the current self-optimizing networking paradigm from meeting the requirements of 5G. We then propose a comprehensive framework for empowering SONs with big data to address the requirements of 5G. Under this framework we first characterize big data in the context of future mobile networks, identifying its sources and future utilities. We then explicate the specific machine learning and data analytics tools that can be exploited to transform big data into the right data that provides a readily useable knowledge base to create end-to-end intelligence of the network. We then explain how a SON engine can build on the dynamic models extractable from the right data. The resultant dynamicity of a big data empowered SON (BSON) makes it more agile and can essentially transform the SON from being a reactive to proactive paradigm and hence act as a key enabler for 5G's extremely low latency requirements. Finally, we demonstrate the key concepts of our proposed BSON framework through a case study of a problem that the classic 3G/4G SON fails to solve.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: Latent Dirichlet Allocation

paper_content:

We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: Design and Realization of Cognitive Routing Resources Using Big Data Analysis in SDN

paper_content:

Future networks need to get knowledge of users' requirements by analyzing big data to achieve intelligence. Open Flow is the first standard communications interface defined between the control and forwarding layers of a SDN architecture. So as to enhance network flexibility and availability, this paper presents a network resource allocation scheme equipped with the ability of user preference awareness, by connecting a cloud platform of analyzing users' data. We use the Hadoop platform to predict the rate and type of the incoming flows and employ them to get the lightest burden link through the LARAC algorithm. It has been verified on the BUPT SDN test bed that this system is capable of allocating the network resources dynamically by predicting the network load in advance. The results show that the proposed methods achieve better load balance than previous networks.
---
paper_title: OpenFlow: enabling innovation in campus networks

paper_content:

This whitepaper proposes OpenFlow: a way for researchers to run experimental protocols in the networks they use every day. OpenFlow is based on an Ethernet switch, with an internal flow-table, and a standardized interface to add and remove flow entries. Our goal is to encourage networking vendors to add OpenFlow to their switch products for deployment in college campus backbones and wiring closets. We believe that OpenFlow is a pragmatic compromise: on one hand, it allows researchers to run experiments on heterogeneous switches in a uniform way at line-rate and with high port-density; while on the other hand, vendors do not need to expose the internal workings of their switches. In addition to allowing researchers to evaluate their ideas in real-world traffic settings, OpenFlow could serve as a useful campus component in proposed large-scale testbeds like GENI. Two buildings at Stanford University will soon run OpenFlow networks, using commercial Ethernet switches and routers. We will work to encourage deployment at other schools; and We encourage you to consider deploying OpenFlow in your university network too
---
paper_title: Design and Realization of Cognitive Routing Resources Using Big Data Analysis in SDN

paper_content:

Future networks need to get knowledge of users' requirements by analyzing big data to achieve intelligence. Open Flow is the first standard communications interface defined between the control and forwarding layers of a SDN architecture. So as to enhance network flexibility and availability, this paper presents a network resource allocation scheme equipped with the ability of user preference awareness, by connecting a cloud platform of analyzing users' data. We use the Hadoop platform to predict the rate and type of the incoming flows and employ them to get the lightest burden link through the LARAC algorithm. It has been verified on the BUPT SDN test bed that this system is capable of allocating the network resources dynamically by predicting the network load in advance. The results show that the proposed methods achieve better load balance than previous networks.
---
paper_title: Pythia: Faster Big Data in Motion through Predictive Software-Defined Network Optimization at Runtime

paper_content:

The rise of Internet of Things sensors, social networking and mobile devices has led to an explosion of available data. Gaining insights into this data has led to the area of Big Data analytics. The MapReduce framework, as implemented in Hadoop, is one of the most popular frameworks for Big Data analysis. To handle the ever-increasing data size, Hadoop is a scalable framework that allows dedicated, seemingly unbound numbers of servers to participate in the analytics process. Response time of an analytics request is an important factor for time to value/insights. While the compute and disk I/O requirements can be scaled with the number of servers, scaling the system leads to increased network traffic. Arguably, the communication-heavy phase of MapReduce contributes significantly to the overall response time, the problem is further aggravated, if communication patterns are heavily skewed, as is not uncommon in many MapReduce workloads. In this paper we present a system that reduces the skew impact by transparently predicting data communication volume at runtime and mapping the many end-to-end flows among the various processes to the underlying network, using emerging software-defined networking technologies to avoid hotspots in the network. Dependent on the network oversubscription ratio, we demonstrate reduction in job completion time between 3% and 46% for popular MapReduce benchmarks like Sort and Nutch.
---
paper_title: Dryad: distributed data-parallel programs from sequential building blocks

paper_content:

Dryad is a general-purpose distributed execution engine for coarse-grain data-parallel applications. A Dryad application combines computational "vertices" with communication "channels" to form a dataflow graph. Dryad runs the application by executing the vertices of this graph on a set of available computers, communicating as appropriate through flies, TCP pipes, and shared-memory FIFOs. The vertices provided by the application developer are quite simple and are usually written as sequential programs with no thread creation or locking. Concurrency arises from Dryad scheduling vertices to run simultaneously on multiple computers, or on multiple CPU cores within a computer. The application can discover the size and placement of data at run time, and modify the graph as the computation progresses to make efficient use of the available resources. Dryad is designed to scale from powerful multi-core single computers, through small clusters of computers, to data centers with thousands of computers. The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
---
paper_title: Routing and Wavelength Assignment in All-Optical Networks

paper_content:

Considers routing connections in a reconfigurable optical network using WDM. Each connection between a pair of nodes in the network is assigned a path through the network and a wavelength on that path, such that connections whose paths share a common link in the network are assigned different wavelengths. The authors derive an upper bound on the carried traffic of connections (or equivalently, a lower bound on the blocking probability) for any routing and wavelength assignment (RWA) algorithm in such a network. The bound scales with the number of wavelengths and is achieved asymptotically (when a large number of wavelengths is available) by a fixed RWA algorithm. The bound can be used as a metric against which the performance of different RWA algorithms can be compared for networks of moderate size. The authors illustrate this by comparing the performance of a simple shortest-path RWA (SP-RWA) algorithm via simulation relative to the bound. They also derive a similar bound for optical networks using dynamic wavelength converters, which are equivalent to circuit-switched telephone networks, and compare the two cases. Finally, they quantify the amount of wavelength reuse achievable in large networks using the SP-RWA via simulation as a function of the number of wavelengths, number of edges, and number of nodes for randomly constructed networks as well as de Bruijn networks. They also quantify the difference in wavelength reuse between two different optical node architectures. >
---
paper_title: “Follow the Sun, Follow the Wind” Lightpath Virtual Topology Reconfiguration in IP Over WDM Network

paper_content:

Green House Gas (GHG) emissions mainly come from the consumption of non-renewable energy. To reduce GHG emissions of IP over WDM networks, we propose to maximize renewable energy usage at each network node location so as to reduce the consumption of non-renewable energy. A “Follow the Sun, Follow the Wind” strategy is proposed for the IP over WDM network to periodically reconfigure the lightpath virtual topology to enable more lightpaths to start or end at nodes where maximum renewable energy is available. We develop a mixed integer linear programming model to design new lightpath virtual topologies. Since the computational complexity of the optimization model is excessive, we also propose a simple but efficient heuristic algorithm to tackle this. Our results indicate that a network operated in this way can significantly reduce non-renewable energy consumption as illustrated in the example network scenarios considered.
---
paper_title: Applying Hadoop Cloud Computing Technique to Optimal Design of Optical Networks (Invited)

paper_content:

We develop a Hadoop cloud computing system to parallel solve the bin-packing optimization problems in the optical network. Simulation studies show that compared to a single machine, the Hadoop system can significantly shorten computation time while achieving performance close to optimality.
---
paper_title: Adaptive FEC selection for lightpaths in elastic optical networks

paper_content:

We propose a new approach to adaptively select FEC types for lightpaths in elastic optical networks. An ILP model and a spectrum-window-based heuristic algorithm are developed to analyze its performance. The proposed FEC selection scheme can achieve good performance with low FEC overhead.
---
paper_title: Distance Adaptive Dynamic Routing and Spectrum Allocation in Elastic Optical Networks With Shared Backup Path Protection

paper_content:

This paper considers distance adaptive dynamic routing and spectrum assignment (RSA) for elastic optical networks with shared backup path protection (SBPP). Efficient heuristic algorithms based on spectrum window planes are proposed for implementing distance and modulation format adaptive RSA so as to maximize spare capacity sharing among multiple protection lightpaths. A differentiated sharable frequency slot cost was also defined for the first time to more efficiently share protection resource. Network performance is evaluated in terms of bandwidth blocking probability (BBP) through simulations. The proposed SBPP technique is effective in reducing BBP and improving spectral efficiency compared to conventional SBPP schemes and 1+1 path protection. The impact of transponder tunability on bandwidth blocking performance is also evaluated to show that a limited tuning range is sufficient to achieve a BBP performance close to that with full tunability.
---
paper_title: Dynamic Virtual Network Reconfiguration Over SDN Orchestrated Multitechnology Optical Transport Domains

paper_content:

Network virtualization is an emerging technique that enables multiple tenants to share an underlying physical infrastructure, isolating the traffic running over different virtual infrastructures/tenants. This technique aims to improve network utilization, while reducing the complexities in terms of network management for operators. Applied to this context, software-defined networking (SDN) paradigm can ease network configurations by enabling network programability and automation, which reduces the amount of operations required from both service and infrastructure providers. SDN techniques are decreasing vendor lock-in issues due to specific configuration methods or protocols. Application-based network operations (ABNO) are a toolbox of key network functional components with the goal of offering application-driven network management. Service provisioning using ABNO may involve direct configuration of data plane elements or delegate it to several control plane modules. We validate the applicability of ABNO to multitenant virtual networks in multitechnology optical domains based on two scenarios, in which multiple control plane instances are orchestrated by the architecture. Congestion detection and failure recovery are chosen to demonstrate fast recalculation and reconfiguration, while hiding the configurations in the physical layer from the upper layer.
---
paper_title: Virtual network topology adaptability based on data analytics for traffic prediction

paper_content:

The introduction of new services requiring large and dynamic bitrate connectivity can cause changes in the direction of the traffic in metro and even core network segments throughout the day. This leads to large overprovisioning in statically managed virtual network topologies (VNTs), which are designed to cope with the traffic forecast. To reduce expenses while ensuring the required grade of service, in this paper we propose a VNT reconfiguration approach based on data analytics for traffic prediction (VENTURE). It regularly reconfigures the VNT based on the predicted traffic, thus adapting the topology to both the current and the predicted traffic volume and direction. A machine learning algorithm based on an artificial neural network is used to provide robust and adaptive traffic models. The reconfiguration problem that takes as its input the traffic prediction is modeled mathematically, and a heuristic is proposed to solve it in practical times. To support VENTURE, we propose an architecture that allows collecting and storing data from monitoring at the routers and that is used to train predictive models for every origin-destination pair. Exhaustive simulation results of the algorithm, together with the experimental assessment of the proposed architecture, are finally presented.
---
paper_title: Your botnet is my botnet: analysis of a botnet takeover

paper_content:

Botnets, networks of malware-infected machines that are controlled by an adversary, are the root cause of a large number of security problems on the Internet. A particularly sophisticated and insidious type of bot is Torpig, a malware program that is designed to harvest sensitive information (such as bank account and credit card data) from its victims. In this paper, we report on our efforts to take control of the Torpig botnet and study its operations for a period of ten days. During this time, we observed more than 180 thousand infections and recorded almost 70 GB of data that the bots collected. While botnets have been "hijacked" and studied previously, the Torpig botnet exhibits certain properties that make the analysis of the data particularly interesting. First, it is possible (with reasonable accuracy) to identify unique bot infections and relate that number to the more than 1.2 million IP addresses that contacted our command and control server. Second, the Torpig botnet is large, targets a variety of applications, and gathers a rich and diverse set of data from the infected victims. This data provides a new understanding of the type and amount of personal information that is stolen by botnets.
---
paper_title: Applied Network Security Monitoring: Collection, Detection, and Analysis

paper_content:

Applied Network Security Monitoringis the essential guide to becoming an NSM analyst from the ground up. This book takes a fundamental approach, complete with real-world examples that teach you the key concepts of NSM. Network security monitoring is based on the principle that prevention eventually fails. In the current threat landscape, no matter how much you try, motivated attackers will eventually find their way into your network. At that point, your ability to detect and respond to that intrusion can be the difference between a small incident and a major disaster. The book follows the three stages of the NSM cycle: collection, detection, and analysis. As you progress through each section, you will have access to insights from seasoned NSM professionals while being introduced to relevant, practical knowledge that you can apply immediately. Discusses the proper methods for planning and executing an NSM data collection strategy Provides thorough hands-on coverage of Snort, Suricata, Bro-IDS, SiLK, PRADS, and more The first book to define multiple analysis frameworks that can be used for performing NSM investigations in a structured and systematic manner Loaded with practical examples that make use of the Security Onion Linux distribution Companion website includes up-to-date blogs from the authors about the latest developments in NSM, complete with supplementary book materials If you've never performed NSM analysis,Applied Network Security Monitoringwill help you grasp the core concepts needed to become an effective analyst. If you are already working in an analysis role, this book will allow you to refine your analytic technique and increase your effectiveness. You will get caught off guard, you will be blind sided, and sometimes you will lose the fight to prevent attackers from accessing your network. This book is about equipping you with the right tools for collecting the data you need, detecting malicious activity, and performingthe analysis that will help you understand the nature of an intrusion. Although prevention can eventually fail, NSM doesn't have to. ** Note: All author royalties from the sale of Applied NSM are being donated to a number of charities selected by the authors.
---
paper_title: Big Data Analytics framework for Peer-to-Peer Botnet detection using Random Forests

paper_content:

Abstract Network traffic monitoring and analysis-related research has struggled to scale for massive amounts of data in real time. Some of the vertical scaling solutions provide good implementation of signature based detection. Unfortunately these approaches treat network flows across different subnets and cannot apply anomaly-based classification if attacks originate from multiple machines at a lower speed, like the scenario of Peer-to-Peer Botnets. In this paper the authors build up on the progress of open source tools like Hadoop, Hive and Mahout to provide a scalable implementation of quasi-real-time intrusion detection system. The implementation is used to detect Peer-to-Peer Botnet attacks using machine learning approach. The contributions of this paper are as follows: (1) Building a distributed framework using Hive for sniffing and processing network traces enabling extraction of dynamic network features; (2) Using the parallel processing power of Mahout to build Random Forest based Decision Tree model which is applied to the problem of Peer-to-Peer Botnet detection in quasi-real-time. The implementation setup and performance metrics are presented as initial observations and future extensions are proposed.
---
paper_title: Performance Evaluation and Optimization of Multi-Dimensional Indexes in Hive

paper_content:

Apache Hive has been widely used for big data processing over large scale clusters by many companies. It provides a declarative query language called HiveQL. The efficiency of filtering out query-irrelevant data from HDFS closely affects the performance of query processing. This is especially true for multi-dimensional, high-selective, and few columns involving queries, which provides sufficient information to reduce the amount of bytes read. Indexing (Compact Index, Aggregate Index, Bitmap Index, DGFIndex, and the index in ORC file) and columnar storage (RCFile, ORC file, and Parquet) are powerful techniques to achieve this. However, it is not trivial to choosing a suitable index and columnar storage based on data and query features. In this paper, we compare the data filtering performance of the above indexes with different columnar storage formats by conducting comprehensive experiments using uniform and skew TPC-H data sets and various multi-dimensional queries, and suggest the best practices of improving multi-dimensional queries in Hive under different conditions.
---
paper_title: Big Data: Tutorial and guidelines on information and process fusion for analytics algorithms with MapReduce

paper_content:

Abstract We live in a world were data are generated from a myriad of sources, and it is really cheap to collect and storage such data. However, the real benefit is not related to the data itself, but with the algorithms that are capable of processing such data in a tolerable elapse time, and to extract valuable knowledge from it. Therefore, the use of Big Data Analytics tools provide very significant advantages to both industry and academia. The MapReduce programming framework can be stressed as the main paradigm related with such tools. It is mainly identified by carrying out a distributed execution for the sake of providing a high degree of scalability, together with a fault-tolerant scheme. In every MapReduce algorithm, first local models are learned with a subset of the original data within the so-called Map tasks. Then, the Reduce task is devoted to fuse the partial outputs generated by each Map. The ways of designing such fusion of information/models may have a strong impact in the quality of the final system. In this work, we will enumerate and analyze two alternative methodologies that may be found both in the specialized literature and in standard Machine Learning libraries for Big Data. Our main objective is to provide an introduction of the characteristics of these methodologies, as well as giving some guidelines for the design of novel algorithms in this field of research. Finally, a short experimental study will allow us to contrast the scalability issues for each type of process fusion in MapReduce for Big Data Analytics.
---
paper_title: B-dids: Mining anomalies in a Big-distributed Intrusion Detection System

paper_content:

The focus of this paper is to present the architecture of a Big-distributed Intrusion Detection System (B-dIDS) to discover multi-pronged attacks which are anomalies existing across multiple subnets in a distributed network. The B-dIDS is composed of two key components: a big data processing engine and an analytics engine. The big data processing is done through HAMR, which is a next generation in-memory MapReduce engine. HAMR has reported high speedups over existing big data solutions across several analytics algorithms. The analytics engine comprises a novel ensemble algorithm, which extracts training data from clusters of the multiple IDS alarms. The clustering is utilized as a preprocessing step to re-label the datasets based on their high similarity to known potential attacks. The overall aim is to predict multi-pronged attacks that are spread across multiple subnets but can be missed if not evaluated in an integrated manner.
---
paper_title: Device Fingerprinting in Wireless Networks: Challenges and Opportunities

paper_content:

Node forgery or impersonation, in which legitimate cryptographic credentials are captured by an adversary, constitutes one major security threat facing wireless networks. The fact that mobile devices are prone to be compromised and reverse engineered significantly increases the risk of such attacks in which adversaries can obtain secret keys on trusted nodes and impersonate the legitimate node. One promising approach toward thwarting these attacks is through the extraction of unique fingerprints that can provide a reliable and robust means for device identification. These fingerprints can be extracted from transmitted signal by analyzing information across the protocol stack. In this paper, the first unified and comprehensive tutorial in the area of wireless device fingerprinting for security applications is presented. In particular, we aim to provide a detailed treatment on developing novel wireless security solutions using device fingerprinting techniques. The objectives are three-fold: (i) to introduce a comprehensive taxonomy of wireless features that can be used in fingerprinting, (ii) to provide a systematic review on fingerprint algorithms including both white-list based and unsupervised learning approaches, and (iii) to identify key open research problems in the area of device fingerprinting and feature extraction, as applied to wireless security.
---
paper_title: A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection

paper_content:

This survey paper describes a focused literature survey of machine learning (ML) and data mining (DM) methods for cyber analytics in support of intrusion detection. Short tutorial descriptions of each ML/DM method are provided. Based on the number of citations or the relevance of an emerging method, papers representing each method were identified, read, and summarized. Because data are so important in ML/DM approaches, some well-known cyber data sets used in ML/DM are described. The complexity of ML/DM algorithms is addressed, discussion of challenges for using ML/DM for cyber security is presented, and some recommendations on when to use a given method are provided.
---
paper_title: Big Data Analytics for User-Activity Analysis and User-Anomaly Detection in Mobile Wireless Network

paper_content:

The next generation wireless networks are expected to operate in fully automated fashion to meet the burgeoning capacity demand and to serve users with superior quality of experience. Mobile wireless networks can leverage spatio-temporal information about user and network condition to embed the system with end-to-end visibility and intelligence. Big data analytics has emerged as a promising approach to unearth meaningful insights and to build artificially intelligent models with assistance of machine learning tools. Utilizing aforementioned tools and techniques, this paper contributes in two ways. First, we utilize mobile network data (Big Data)—call detail record—to analyze anomalous behavior of mobile wireless network. For anomaly detection purposes, we use unsupervised clustering techniques namely k-means clustering and hierarchical clustering. We compare the detected anomalies with ground truth information to verify their correctness. From the comparative analysis, we observe that when the network experiences abruptly high (unusual) traffic demand at any location and time, it identifies that as anomaly. This helps in identifying regions of interest in the network for special action such as resource allocation, fault avoidance solution, etc. Second, we train a neural-network-based prediction model with anomalous and anomaly-free data to highlight the effect of anomalies in data while training/building intelligent models. In this phase, we transform our anomalous data to anomaly-free and we observe that the error in prediction, while training the model with anomaly-free data has largely decreased as compared to the case when the model was trained with anomalous data.
---
paper_title: Big data-driven optimization for mobile networks toward 5G

paper_content:

Big data offers a plethora of opportunities to mobile network operators for improving quality of service. This article explores various means of integrating big data analytics with network optimization toward the objective of improving the user quality of experience. We first propose a framework of Big Data-Driven (BDD) mobile network optimization. We then present the characteristics of big data that are collected not only from user equipment but also from mobile networks. Moreover, several techniques in data collection and analytics are discussed from the viewpoint of network optimization. Certain user cases on the application of the proposed framework for improving network performance are also given in order to demonstrate the feasibility of the framework. With the integration of the emerging fifth generation (5G) mobile networks with big data analytics, the quality of our daily mobile life is expected to be tremendously enhanced.
---
paper_title: Monitoring and analyzing big traffic data of a large-scale cellular network with Hadoop

paper_content:

Network traffic monitoring and analysis is of theoretical and practical significance for optimizing network resource and improving user experience. However, existing solutions, which usually rely on a high-performance server with large storage capacity, are not scalable for detailed analysis of a large volume of traffic data. In this article, we present a traffic monitoring and analysis system for large-scale networks based on Hadoop, an open-source distributed computing platform for big data processing on commodity hardware. This system has been deployed in the core network of a large cellular network and extensively evaluated. The results demonstrate that the system can efficiently processes 4.2 Tbytes of traffic data from 123 Gb/s links with high performance and low cost every day.
---
paper_title: Anomaly Detection In Cellular Network Data Using Big Data Analytics

paper_content:

Anomaly detection is a key component in which perturbations from a normal behavior suggests a misconfigured/ mismatched data in related systems. In this paper, we present a call detail record based anomaly detection method (CADM) that analyzes the users's calling activities and detects the abnormal behavior of user movements in a real cellular network. CADM is capable of detecting the location of the site that an anomaly has occurred. We evaluate the proposed CADM by performing experiments over the call-detail records of AVEA, a mobile service provider in Turkey.
---
paper_title: Using Big Data to Improve Customer Experience and Business Performance

paper_content:

In an environment where communications service providers (CSPs) increasingly have the same service offers and devices, offering a superior customer experience is a priority to compete. Solutions that have the ability to highlight what really matters in driving customer satisfaction and deliver actionable insights from their wide-reaching customer, network, and service data are key differentiators for CSPs. This paper explores ways of integrating big data insights with automated and assisted processes related to key customer touchpoints to ultimately improve the customer experience. We show how innovation from Alcatel-Lucent and Bell Labs helps CSPs improve their business performance, using unique methodology designed to select the right key quality indicators, build accurate key business objective “formula,” predict customer behavior, and ultimately understand which factors are influencing the most. This can be used for example to improve the Net Promoter Score (NPS). The net result is a happier customer and a higher customer value.
---
paper_title: Mobile Internet big data platform in China Unicom

paper_content:

China Unicom, the largest WCDMA 3G operator in China, meets the requirements of the historical Mobile Internet Explosion, or the surging of Mobile Internet Traffic from mobile terminals. According to the internal statistics of China Unicom, mobile user traffic has increased rapidly with a Compound Annual Growth Rate (CAGR) of 135%. Currently China Unicom monthly stores more than 2 trillion records, data volume is over 525 TB, and the highest data volume has reached a peak of 5 PB. Since October 2009, China Unicom has been developing a home-brewed big data storage and analysis platform based on the open source Hadoop Distributed File System (HDFS) as it has a long-term strategy to make full use of this Big Data. All Mobile Internet Traffic is well served using this big data platform. Currently, the writing speed has reached 1 390 000 records per second, and the record retrieval time in the table that contains trillions of records is less than 100 ms. To take advantage of this opportunity to be a Big Data Operator, China Unicom has developed new functions and has multiple innovations to solve space and time constraint challenges presented in data processing. In this paper, we will introduce our big data platform in detail. Based on this big data platform, China Unicom is building an industry ecosystem based on Mobile Internet Big Data, and considers that a telecom operator centric ecosystem can be formed that is critical to reach prosperity in the modern communications business.
---
paper_title: Operational Experiences with Anomaly Detection in Backbone Networks

paper_content:

Although network security is a crucial aspect for network operators, there are still very few works that have examined the anomalies present in large backbone networks and evaluated the performance of existing anomaly detection solutions in operational environments. The objective of this work is to fill this gap by reporting hands-on experience in the evaluation and deployment of an anomaly detection solution for the GEANT backbone network. During this process, we analyzed three different commercial tools for anomaly detection and then deployed one of them for several months in the 18 points-of-presence of GEANT. We first explain the general requirements that an anomaly detection system should satisfy from the point of view of a network operator. Afterwards, we describe the evaluation of the tools and present a study of the anomalies found in a continental backbone network after operationally using the finally deployed tool for half a year. We think that this first hand information can be of great interest to both professionals and researchers working on network security and can also guide future research towards more practical problems faced by network operators.
---
paper_title: Traffic monitoring and analysis for the optimization of a 3G network

paper_content:

Recent years have recorded a surge of research activities on IP traffic monitoring, enabled by the availability of monitoring hardware and large-scale storage at accessible costs. More recently, passive monitoring has been applied to operational 3G networks. The passive observation of network traffic, coupled with advanced traffic-analysis methods, can be a powerful and cost-effective means to infer the network status and localize points of performance degradation without requiring complete access to all network elements. Furthermore, the availability of high-quality traces can be exploited to predict the load of the network under hypothetical conditions, variations of the actual network configuration at the capturing time. Both approaches can be useful for some engineering and reoptimization tasks that are commonly encountered in the lifetime of an operational 3G network. In abstract terms, the availability of high-quality traces can greatly empower the measurement-based optimization cycle, with human experts in the loop, thus driving an already operational 3G network toward improved performances. In this article we discuss the contribution that traffic monitoring and analysis (TMA) can provide to the optimization of an operational 3G network
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: Big data-driven optimization for mobile networks toward 5G

paper_content:

Big data offers a plethora of opportunities to mobile network operators for improving quality of service. This article explores various means of integrating big data analytics with network optimization toward the objective of improving the user quality of experience. We first propose a framework of Big Data-Driven (BDD) mobile network optimization. We then present the characteristics of big data that are collected not only from user equipment but also from mobile networks. Moreover, several techniques in data collection and analytics are discussed from the viewpoint of network optimization. Certain user cases on the application of the proposed framework for improving network performance are also given in order to demonstrate the feasibility of the framework. With the integration of the emerging fifth generation (5G) mobile networks with big data analytics, the quality of our daily mobile life is expected to be tremendously enhanced.
---
paper_title: A dynamic bandwidth allocation algorithm in mobile networks with big data of users and networks

paper_content:

Data collection has become easy due to the rapid development of both mobile devices and wireless networks. In each second, numerous data are generated by user devices and collected through wireless networks. These data, carrying user and network related information, are invaluable for network management. However, they were seldom employed to improve network performance in existing research work. In this article we propose a bandwidth allocation algorithm to increase the throughput of cellular network users by exploring user and network data collected from user devices. With the aid of these data, users can be categorized into clusters and share bandwidth to improve the resource utilization of the network. Simulation results indicate that the proposed scheme is able to rationally form clusters among mobile users and thus significantly increase the throughput and bandwidth efficiency of the network.
---
paper_title: Improving Data Extraction Efficiency of Cache Nodes in Cognitive Radio Networks Using Big Data Analysis

paper_content:

In cognitive radio networks, unlicensed users are allowed to use underutilized licensed spectrum until licensed users' transmission quality of service is not compromised. As soon as the conflict goes beyond a certain limit, SU must leave the spectrum and move to the other nearby free band. At the time of interruption, sensing the nearby free channels and switching to them will take some time, hence the ongoing data will be interrupted, which will delay the data transmission. To minimize this delay, creating cache of the SU signal at multiple nodes in a cluster has shown significant improvement in reducing the transmission delay if cache placement is done systematically. This systematic and accurate placement of cache is possible if the data accumulated is accessed and processed quickly. Taking into account the vastness of cluster networks, a huge amount of data will be required to be accessed and processed. Cognitive Radio networks are very complex structures when it comes to the information sharing amongst the secondary users and with the cluster head. Taking into account, whether unlicensed users share their information with other secondary users, and in case if they do, how much proportion of it they allow the fusion center to process, several big data scenarios exist. This paper discusses the possible information sharing scenarios in cognitive radio network systems and their possible Big Data Solutions.
---
paper_title: On Big Data Analytics for Greener and Softer RAN

paper_content:

Big data analytics applied to signaling, traffic, and wireless environment data in mobile communication networks can help realize autonomous network optimization and build big data-based network operation. In this paper, a signaling-based intelligent network optimization scheme is introduced and applied to the current mobile communication networks, such as 4G Long Term Evolution. In 5G era, big data analytics can help mine user and service requirements from the radio access network level, thus allowing a more efficient 5G design and operation. This paper illustrates how it would significantly facilitate local content provision, dynamical network and functionality deployment, user behavior awareness, fine-tuned network operation, and globally optimized energy saving solutions. It is anticipated that the big data-based 5G network design, and the operation will be greener and softer, and better meet the ever increasing user-centric requirements of mobile communication.
---
paper_title: Design and Realization of Cognitive Routing Resources Using Big Data Analysis in SDN

paper_content:

Future networks need to get knowledge of users' requirements by analyzing big data to achieve intelligence. Open Flow is the first standard communications interface defined between the control and forwarding layers of a SDN architecture. So as to enhance network flexibility and availability, this paper presents a network resource allocation scheme equipped with the ability of user preference awareness, by connecting a cloud platform of analyzing users' data. We use the Hadoop platform to predict the rate and type of the incoming flows and employ them to get the lightest burden link through the LARAC algorithm. It has been verified on the BUPT SDN test bed that this system is capable of allocating the network resources dynamically by predicting the network load in advance. The results show that the proposed methods achieve better load balance than previous networks.
---
paper_title: Pythia: Faster Big Data in Motion through Predictive Software-Defined Network Optimization at Runtime

paper_content:

The rise of Internet of Things sensors, social networking and mobile devices has led to an explosion of available data. Gaining insights into this data has led to the area of Big Data analytics. The MapReduce framework, as implemented in Hadoop, is one of the most popular frameworks for Big Data analysis. To handle the ever-increasing data size, Hadoop is a scalable framework that allows dedicated, seemingly unbound numbers of servers to participate in the analytics process. Response time of an analytics request is an important factor for time to value/insights. While the compute and disk I/O requirements can be scaled with the number of servers, scaling the system leads to increased network traffic. Arguably, the communication-heavy phase of MapReduce contributes significantly to the overall response time, the problem is further aggravated, if communication patterns are heavily skewed, as is not uncommon in many MapReduce workloads. In this paper we present a system that reduces the skew impact by transparently predicting data communication volume at runtime and mapping the many end-to-end flows among the various processes to the underlying network, using emerging software-defined networking technologies to avoid hotspots in the network. Dependent on the network oversubscription ratio, we demonstrate reduction in job completion time between 3% and 46% for popular MapReduce benchmarks like Sort and Nutch.
---
paper_title: Virtual network topology adaptability based on data analytics for traffic prediction

paper_content:

The introduction of new services requiring large and dynamic bitrate connectivity can cause changes in the direction of the traffic in metro and even core network segments throughout the day. This leads to large overprovisioning in statically managed virtual network topologies (VNTs), which are designed to cope with the traffic forecast. To reduce expenses while ensuring the required grade of service, in this paper we propose a VNT reconfiguration approach based on data analytics for traffic prediction (VENTURE). It regularly reconfigures the VNT based on the predicted traffic, thus adapting the topology to both the current and the predicted traffic volume and direction. A machine learning algorithm based on an artificial neural network is used to provide robust and adaptive traffic models. The reconfiguration problem that takes as its input the traffic prediction is modeled mathematically, and a heuristic is proposed to solve it in practical times. To support VENTURE, we propose an architecture that allows collecting and storing data from monitoring at the routers and that is used to train predictive models for every origin-destination pair. Exhaustive simulation results of the algorithm, together with the experimental assessment of the proposed architecture, are finally presented.
---
paper_title: Applying Hadoop Cloud Computing Technique to Optimal Design of Optical Networks (Invited)

paper_content:

We develop a Hadoop cloud computing system to parallel solve the bin-packing optimization problems in the optical network. Simulation studies show that compared to a single machine, the Hadoop system can significantly shorten computation time while achieving performance close to optimality.
---
paper_title: Anomaly Detection In Cellular Network Data Using Big Data Analytics

paper_content:

Anomaly detection is a key component in which perturbations from a normal behavior suggests a misconfigured/ mismatched data in related systems. In this paper, we present a call detail record based anomaly detection method (CADM) that analyzes the users's calling activities and detects the abnormal behavior of user movements in a real cellular network. CADM is capable of detecting the location of the site that an anomaly has occurred. We evaluate the proposed CADM by performing experiments over the call-detail records of AVEA, a mobile service provider in Turkey.
---
paper_title: Device Fingerprinting in Wireless Networks: Challenges and Opportunities

paper_content:

Node forgery or impersonation, in which legitimate cryptographic credentials are captured by an adversary, constitutes one major security threat facing wireless networks. The fact that mobile devices are prone to be compromised and reverse engineered significantly increases the risk of such attacks in which adversaries can obtain secret keys on trusted nodes and impersonate the legitimate node. One promising approach toward thwarting these attacks is through the extraction of unique fingerprints that can provide a reliable and robust means for device identification. These fingerprints can be extracted from transmitted signal by analyzing information across the protocol stack. In this paper, the first unified and comprehensive tutorial in the area of wireless device fingerprinting for security applications is presented. In particular, we aim to provide a detailed treatment on developing novel wireless security solutions using device fingerprinting techniques. The objectives are three-fold: (i) to introduce a comprehensive taxonomy of wireless features that can be used in fingerprinting, (ii) to provide a systematic review on fingerprint algorithms including both white-list based and unsupervised learning approaches, and (iii) to identify key open research problems in the area of device fingerprinting and feature extraction, as applied to wireless security.
---
paper_title: Deep Network Analyzer (DNA): A Big Data Analytics Platform for Cellular Networks

paper_content:

In this paper, we present deep network analyzer (DNA), a big data analytics platform for anomaly detection (AD) and root cause analysis (RCA) in mobile wireless networks. DNA is motivated by the growing scale and complexity of cellular networks along with the lack of advanced big data analytics tools for effective network management. It abstracts the RCA process into two modules, namely rule (fingerprint) learning and the module of AD and fingerprint matching. We first develop a rare association rule mining method to learn the symptoms of network anomalies and to build a fingerprint knowledge database from the historic data. Then a statistical machine learning approach is employed to identify the anomalies within the incoming dataset collected via various probes in the network and map the fingerprints of the detected anomalies to the rules in the knowledge database. The DNA platform has been tested using the real production data from the field and has been shown to be a highly effective platform for AD and RCA for large-scale cellular systems serving tens of millions of mobile users.
---
paper_title: Big Data Analytics for User-Activity Analysis and User-Anomaly Detection in Mobile Wireless Network

paper_content:

The next generation wireless networks are expected to operate in fully automated fashion to meet the burgeoning capacity demand and to serve users with superior quality of experience. Mobile wireless networks can leverage spatio-temporal information about user and network condition to embed the system with end-to-end visibility and intelligence. Big data analytics has emerged as a promising approach to unearth meaningful insights and to build artificially intelligent models with assistance of machine learning tools. Utilizing aforementioned tools and techniques, this paper contributes in two ways. First, we utilize mobile network data (Big Data)—call detail record—to analyze anomalous behavior of mobile wireless network. For anomaly detection purposes, we use unsupervised clustering techniques namely k-means clustering and hierarchical clustering. We compare the detected anomalies with ground truth information to verify their correctness. From the comparative analysis, we observe that when the network experiences abruptly high (unusual) traffic demand at any location and time, it identifies that as anomaly. This helps in identifying regions of interest in the network for special action such as resource allocation, fault avoidance solution, etc. Second, we train a neural-network-based prediction model with anomalous and anomaly-free data to highlight the effect of anomalies in data while training/building intelligent models. In this phase, we transform our anomalous data to anomaly-free and we observe that the error in prediction, while training the model with anomaly-free data has largely decreased as compared to the case when the model was trained with anomalous data.
---
paper_title: Using Big Data to Improve Customer Experience and Business Performance

paper_content:

In an environment where communications service providers (CSPs) increasingly have the same service offers and devices, offering a superior customer experience is a priority to compete. Solutions that have the ability to highlight what really matters in driving customer satisfaction and deliver actionable insights from their wide-reaching customer, network, and service data are key differentiators for CSPs. This paper explores ways of integrating big data insights with automated and assisted processes related to key customer touchpoints to ultimately improve the customer experience. We show how innovation from Alcatel-Lucent and Bell Labs helps CSPs improve their business performance, using unique methodology designed to select the right key quality indicators, build accurate key business objective “formula,” predict customer behavior, and ultimately understand which factors are influencing the most. This can be used for example to improve the Net Promoter Score (NPS). The net result is a happier customer and a higher customer value.
---
paper_title: Anomaly Detection In Cellular Network Data Using Big Data Analytics

paper_content:

Anomaly detection is a key component in which perturbations from a normal behavior suggests a misconfigured/ mismatched data in related systems. In this paper, we present a call detail record based anomaly detection method (CADM) that analyzes the users's calling activities and detects the abnormal behavior of user movements in a real cellular network. CADM is capable of detecting the location of the site that an anomaly has occurred. We evaluate the proposed CADM by performing experiments over the call-detail records of AVEA, a mobile service provider in Turkey.
---
paper_title: Deep Network Analyzer (DNA): A Big Data Analytics Platform for Cellular Networks

paper_content:

In this paper, we present deep network analyzer (DNA), a big data analytics platform for anomaly detection (AD) and root cause analysis (RCA) in mobile wireless networks. DNA is motivated by the growing scale and complexity of cellular networks along with the lack of advanced big data analytics tools for effective network management. It abstracts the RCA process into two modules, namely rule (fingerprint) learning and the module of AD and fingerprint matching. We first develop a rare association rule mining method to learn the symptoms of network anomalies and to build a fingerprint knowledge database from the historic data. Then a statistical machine learning approach is employed to identify the anomalies within the incoming dataset collected via various probes in the network and map the fingerprints of the detected anomalies to the rules in the knowledge database. The DNA platform has been tested using the real production data from the field and has been shown to be a highly effective platform for AD and RCA for large-scale cellular systems serving tens of millions of mobile users.
---
paper_title: Spatial big data and wireless networks: experiences, applications, and research challenges

paper_content:

In this article we demonstrate that spatial big data can play a key role in many emerging wireless networking applications. We also argue that spatial and spatiotemporal problems have their own very distinct role in the big data context compared to the commonly considered relational problems. We describe three major application scenarios for spatial big data, each imposing specific design and research challenges. We then present our work on developing highly scalable parallel processing frameworks for spatial data in the Hadoop framework using the MapReduce computational model. Our results show that using Hadoop enables highly scalable implementations of algorithms for common spatial data processing problems. However, development of these implementations requires significant specialized knowledge, demonstrating the need for development of more user-friendly alternatives.
---
paper_title: Internet of Things in Industries: A Survey

paper_content:

Internet of Things (IoT) has provided a promising opportunity to build powerful industrial systems and applications by leveraging the growing ubiquity of radio-frequency identification (RFID), and wireless, mobile, and sensor devices. A wide range of industrial IoT applications have been developed and deployed in recent years. In an effort to understand the development of IoT in industries, this paper reviews the current research of IoT, key enabling technologies, major IoT applications in industries, and identifies research trends and challenges. A main contribution of this review paper is that it summarizes the current state-of-the-art IoT in industries systematically.
---
paper_title: Understanding individual human mobility patterns

paper_content:

The mapping of large-scale human movements is important for urban planning, traffic forecasting and epidemic prevention. Work in animals had suggested that their foraging might be explained in terms of a random walk, a mathematical rendition of a series of random steps, or a Levy flight, a random walk punctuated by occasional larger steps. The role of Levy statistics in animal behaviour is much debated — as explained in an accompanying News Feature — but the idea of extending it to human behaviour was boosted by a report in 2006 of Levy flight-like patterns in human movement tracked via dollar bills. A new human study, based on tracking the trajectory of 100,000 cell-phone users for six months, reveals behaviour close to a Levy pattern, but deviating from it as individual trajectories show a high degree of temporal and spatial regularity: work and other commitments mean we are not as free to roam as a foraging animal. But by correcting the data to accommodate individual variation, simple and predictable patterns in human travel begin to emerge. The cover photo (by Cesar Hidalgo) captures human mobility in New York's Grand Central Station. This study used a sample of 100,000 mobile phone users whose trajectory was tracked for six months to study human mobility patterns. Displacements across all users suggest behaviour close to the Levy-flight-like pattern observed previously based on the motion of marked dollar bills, but with a cutoff in the distribution. The origin of the Levy patterns observed in the aggregate data appears to be population heterogeneity and not Levy patterns at the level of the individual. Despite their importance for urban planning1, traffic forecasting2 and the spread of biological3,4,5 and mobile viruses6, our understanding of the basic laws governing human motion remains limited owing to the lack of tools to monitor the time-resolved location of individuals. Here we study the trajectory of 100,000 anonymized mobile phone users whose position is tracked for a six-month period. We find that, in contrast with the random trajectories predicted by the prevailing Levy flight and random walk models7, human trajectories show a high degree of temporal and spatial regularity, each individual being characterized by a time-independent characteristic travel distance and a significant probability to return to a few highly frequented locations. After correcting for differences in travel distances and the inherent anisotropy of each trajectory, the individual travel patterns collapse into a single spatial probability distribution, indicating that, despite the diversity of their travel history, humans follow simple reproducible patterns. This inherent similarity in travel patterns could impact all phenomena driven by human mobility, from epidemic prevention to emergency response, urban planning and agent-based modelling.
---
paper_title: Challenges and Opportunities with Big Data

paper_content:

The promise of data-driven decision-making is now being recognized broadly, and there is growing enthusiasm for the notion of "Big Data," including the recent announcement from the White House about new funding initiatives across different agencies, that target research for Big Data. While the promise of Big Data is real -- for example, it is estimated that Google alone contributed 54 billion dollars to the US economy in 2009 -- there is no clear consensus on what is Big Data. In fact, there have been many controversial statements about Big Data, such as "Size is the only thing that matters." In this panel we will try to explore the controversies and debunk the myths surrounding Big Data.
---
paper_title: Challenges of Privacy Protection in Big Data Analytics

paper_content:

The big data paradigm implies that almost every type of information eventually can be derived from sufficiently large datasets. However, in such terms, linkage of personal data of individuals poses a severe threat to privacy and civil rights. In this position paper, we propose a set of challenges that have to be addressed in order to perform big data analytics in a privacy-compliant way.
---
paper_title: Six Provocations for Big Data

paper_content:

The era of Big Data has begun. Computer scientists, physicists, economists, mathematicians, political scientists, bio-informaticists, sociologists, and many others are clamoring for access to the massive quantities of information produced by and about people, things, and their interactions. Diverse groups argue about the potential benefits and costs of analyzing information from Twitter, Google, Verizon, 23andMe, Facebook, Wikipedia, and every space where large groups of people leave digital traces and deposit data. Significant questions emerge. Will large-scale analysis of DNA help cure diseases? Or will it usher in a new wave of medical inequality? Will data analytics help make people’s access to information more efficient and effective? Or will it be used to track protesters in the streets of major cities? Will it transform how we study human communication and culture, or narrow the palette of research options and alter what ‘research’ means? Some or all of the above?This essay offers six provocations that we hope can spark conversations about the issues of Big Data. Given the rise of Big Data as both a phenomenon and a methodological persuasion, we believe that it is time to start critically interrogating this phenomenon, its assumptions, and its biases.(This paper was presented at Oxford Internet Institute’s “A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society” on September 21, 2011.)
---
paper_title: Toward Scalable Systems for Big Data Analytics: A Technology Tutorial

paper_content:

Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.
---
paper_title: Big Data security and privacy: A review

paper_content:

While Big Data gradually become a hot topic of research and business and has been everywhere used in many industries, Big Data security and privacy has been increasingly concerned. However, there is an obvious contradiction between Big Data security and privacy and the widespread use of Big Data. In this paper, we firstly reviewed the enormous benefits and challenges of security and privacy in Big Data. Then, we present some possible methods and techniques to ensure Big Data security and privacy.
---
paper_title: Big data analytics: a survey

paper_content:

The age of big data is now coming. But the traditional data ::: analytics may not be able to handle such large quantities of data. The question that arises now is, how to develop a high performance platform to efficiently analyze big data and ::: how to design an appropriate mining algorithm to find the useful things from big data. To deeply discuss this issue, this paper begins with a brief introduction to data analytics, followed by the discussions of big data analytics. ::: Some important open issues and further research directions will also be presented for the next step of big data analytics.
---
paper_title: Trends in big data analytics

paper_content:

Abstract One of the major applications of future generation parallel and distributed systems is in big-data analytics. Data repositories for such applications currently exceed exabytes and are rapidly increasing in size. Beyond their sheer magnitude, these datasets and associated applications’ considerations pose significant challenges for method and software development. Datasets are often distributed and their size and privacy considerations warrant distributed techniques. Data often resides on platforms with widely varying computational and network capabilities. Considerations of fault-tolerance, security, and access control are critical in many applications (Dean and Ghemawat, 2004; Apache hadoop). Analysis tasks often have hard deadlines, and data quality is a major concern in yet other applications. For most emerging applications, data-driven models and methods, capable of operating at scale, are as-yet unknown. Even when known methods can be scaled, validation of results is a major issue. Characteristics of hardware platforms and the software stack fundamentally impact data analytics. In this article, we provide an overview of the state-of-the-art and focus on emerging trends to highlight the hardware, software, and application landscape of big-data analytics.
---
paper_title: Toward Scalable Systems for Big Data Analytics: A Technology Tutorial

paper_content:

Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.
---
paper_title: Big Data: A Survey

paper_content:

In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.
---
paper_title: Maximizing Air Gap and Efficiency of Magnetic Resonant Coupling for Wireless Power Transfer Using Equivalent Circuit and Neumann Formula

paper_content:

The progress in the field of wireless power transfer in the last few years is remarkable. With recent research, transferring power across large air gaps has been achieved. Both small and large electric equipment have been proposed, e.g., wireless power transfer for small equipment (mobile phones and laptops) and for large equipment (electric vehicles). Furthermore, replacing every cord with wireless power transfer is proposed. The coupled mode theory was proposed in 2006 and proven in 2007. Magnetic and electric resonant couplings allow power to traverse large air gaps with high efficiency. This technology is closely related to electromagnetic induction and has been applied to antennas and resonators used for filters in communication technology. We have studied these phenomena and technologies using equivalent circuits, which is a more familiar format for electrical engineers than the coupled mode theory. In this paper, we analyzed the relationship between maximum efficiency air gap using equivalent circuits and the Neumann formula and proposed equations for the conditions required to achieve maximum efficiency for a given air gap. The results of these equations match well with the results of electromagnetic field analysis and experiments.
---
paper_title: Big Data: A Survey

paper_content:

In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.
---

Title: Big Data Analytics for Wireless and Wired Network Design: A Survey
Section 1: Introduction
Description 1: Discuss the significance of big data in network design, define big data, and introduce the characteristics and challenges of big data.

Section 2: Case studies of the use of big data analytics for wireless and wired networks.
Description 2: Present various case studies that demonstrate the application of big data analytics in wireless and wired network scenarios.

Section 3: Role of big data analytics in cellular network design
Description 3: Review research on the use of big data analytics for the design and optimization of cellular networks.

Section 4: The role of big data analytics in SDN & intra-data center networks
Description 4: Analyze the employment of big data analytics in Software Defined Networks (SDN) and intra-data center networks for traffic prediction and reduction.

Section 5: The Role of Big Data Analytics in Optical Networks
Description 5: Discuss research papers that use big data analytics for optical network optimization and traffic prediction.

Section 6: Network Security
Description 6: Explore how big data analytics is used to enhance network security, including threat detection and device fingerprinting.

Section 7: Big data analytics in the industry
Description 7: Highlight commercial solutions offered by various companies that leverage big data analytics for network optimization and management.

Section 8: Big data analytics-powered design cycle and challenges
Description 8: Describe the network design cycle powered by big data analytics and discuss challenges such as security, privacy, data center scalability, and more.

Section 9: Open directions for future research
Description 9: Identify potential research directions and areas that require further exploration in the realm of big data analytics for network design.

Section 10: Conclusions
Description 10: Summarize key findings, emphasize the importance of big data analytics in network design, and predict future trends and advancements.