Big Data Reduction Methods: A Survey
---
paper_title: Tradeoffs in scalable data routing for deduplication clusters

paper_content:

As data have been growing rapidly in data centers, deduplication storage systems continuously face challenges in providing the corresponding throughputs and capacities necessary to move backup data within backup and recovery window times. One approach is to build a cluster deduplication storage system with multiple deduplication storage system nodes. The goal is to achieve scalable throughput and capacity using extremely high-throughput (e.g. 1.5 GB/s) nodes, with a minimal loss of compression ratio. The key technical issue is to route data intelligently at an appropriate granularity. ::: ::: We present a cluster-based deduplication system that can deduplicate with high throughput, support deduplication ratios comparable to that of a single system, and maintain a low variation in the storage utilization of individual nodes. In experiments with dozens of nodes, we examine tradeoffs between stateless data routing approaches with low overhead and stateful approaches that have higher overhead but avoid imbalances that can adversely affect deduplication effectiveness for some datasets in large clusters. The stateless approach has been deployed in a two-node commercial system that achieves 3 GB/s for multi-stream deduplication throughput and currently scales to 5.6 PB of storage (assuming 20X total compression).
---
paper_title: Towards Network Reduction on Big Data

paper_content:

The increasing ease of data collection experience and the increasing availability of large data storage space lead to the existence of very large datasets that are commonly referred as "Big Data". Such data not only take over large amount of database storage, but also increase the difficulties for data analysis due to data diversity, which, also makes the datasets seemingly isolated with each other. In this paper, we present a solution to the problem that is to build up connections among the diverse datasets, based upon their similarities. Particularly, a concept of similarity graph along with a similarity graph generation algorithm were introduced. We then proposed a similarity graph reduction algorithm that reduces vertices of the graph for the purpose of graph simplification.
---
paper_title: Challenges of Big Data Analysis

paper_content:

Big Data bring new opportunities to modern society and challenges to data scientists. On the one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This paper gives overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasize on the viability of the sparsest solution in high-confidence set and point out that exogenous assumptions in most statistical methods for Big Data cannot be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.
---
paper_title: Characterizing the efficiency of data deduplication for big data storage management

paper_content:

The demand for data storage and processing is increasing at a rapid speed in the big data era. Such a tremendous amount of data pushes the limit on storage capacity and on the storage network. A significant portion of the dataset in big data workloads is redundant. As a result, deduplication technology, which removes replicas, becomes an attractive solution to save disk space and traffic in a big data environment. However, the overhead of extra CPU computation (hash indexing) and IO latency introduced by deduplication should be considered. Therefore, the net effect of using deduplication for big data workloads needs to be examined. To this end, we characterize the redundancy of typical big data workloads to justify the need for deduplication. We analyze and characterize the performance and energy impact brought by deduplication under various big data environments. In our experiments, we identify three sources of redundancy in big data workloads: 1) deploying more nodes, 2) expanding the dataset, and 3) using replication mechanisms. We elaborate on the advantages and disadvantages of different deduplication layers, locations, and granularities. In addition, we uncover the relation between energy overhead and the degree of redundancy. Furthermore, we investigate the deduplication efficiency in an SSD environment for big data workloads.
---
paper_title: Computing persistent features in big data: A distributed dimension reduction approach

paper_content:

Persistent homology has become one of the most popular tools used in topological data analysis for analyzing big data sets. In an effort to minimize the computational complexity of finding the persistent homology of a data set, we develop a simplicial collapse algorithm called the selective collapse. This algorithm works by representing the previously developed strong collapse as a forest and uses that forest data to improve the speed of both the strong collapse and of persistent homology. Finally, we demonstrate the savings in computational complexity using geometric random graphs.
---
paper_title: Silo: A similarity-locality based near-exact deduplication scheme with low ram overhead and high throughput

paper_content:

Data Deduplication is becoming increasingly popular in storage systems as a space-efficient approach to data backup and archiving. Most existing state-of-the-art deduplication methods are either locality based or similarity based, which, according to our analysis, do not work adequately in many situations. While the former produces poor deduplication throughput when there is little or no locality in datasets, the latter can fail to identify and thus remove significant amounts of redundant data when there is a lack of similarity among files. In this paper, we present SiLo, a near-exact deduplication system that effectively and complementarily exploits similarity and locality to achieve high duplicate elimination and throughput at extremely low RAM overheads. The main idea behind SiLo is to expose and exploit more similarity by grouping strongly correlated small files into a segment and segmenting large files, and to leverage locality in the backup stream by grouping contiguous segments into blocks to capture similar and duplicate data missed by the probabilistic similarity detection. By judiciously enhancing similarity through the exploitation of locality and vice versa, the SiLo approach is able to significantly reduce RAM usage for index-lookup and maintain a very high deduplication throughput. Our experimental evaluation of SiLo based on real-world datasets shows that the SiLo system consistently and significantly outperforms two existing state-of-the-art system, one based on similarity and the other based on locality, under various workload conditions.
---
paper_title: Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis

paper_content:

Higher-order tensors and their decompositions are abundantly present in domains such as signal processing (e.g., higher-order statistics [1] and sensor array processing [2]), scientific computing (e.g., discretized multivariate functions [3]?[6]), and quantum information theory (e.g., representation of quantum many-body states [7]). In many applications, the possibly huge tensors can be approximated well by compact multilinear models or decompositions. Tensor decompositions are more versatile tools than the linear models resulting from traditional matrix approaches. Compared to matrices, tensors have at least one extra dimension. The number of elements in a tensor increases exponentially with the number of dimensions, and so do the computational and memory requirements. The exponential dependency (and the problems that are caused by it) is called the curse of dimensionality. The curse limits the order of the tensors that can be handled. Even for a modest order, tensor problems are often large scale. Large tensors can be handled, and the curse can be alleviated or even removed by using a decomposition that represents the tensor instead of using the tensor itself. However, most decomposition algorithms require full tensors, which renders these algorithms infeasible for many data sets. If a tensor can be represented by a decomposition, this hypothesized structure can be exploited by using compressed sensing (CS) methods working on incomplete tensors, i.e., tensors with only a few known elements.
---
paper_title: Analyzing Big Data with Dynamic Quantum Clustering

paper_content:

How does one search for a needle in a multi-dimensional haystack without knowing what a needle is and without knowing if there is one in the haystack? This kind of problem requires a paradigm shift - away from hypothesis driven searches of the data - towards a methodology that lets the data speak for itself. Dynamic Quantum Clustering (DQC) is such a methodology. DQC is a powerful visual method that works with big, high-dimensional data. It exploits variations of the density of the data (in feature space) and unearths subsets of the data that exhibit correlations among all the measured variables. The outcome of a DQC analysis is a movie that shows how and why sets of data-points are eventually classified as members of simple clusters or as members of - what we call - extended structures. This allows DQC to be successfully used in a non-conventional exploratory mode where one searches data for unexpected information without the need to model the data. We show how this works for big, complex, real-world datasets that come from five distinct fields: i.e., x-ray nano-chemistry, condensed matter, biology, seismology and finance. These studies show how DQC excels at uncovering unexpected, small - but meaningful - subsets of the data that contain important information. We also establish an important new result: namely, that big, complex datasets often contain interesting structures that will be missed by many conventional clustering techniques. Experience shows that these structures appear frequently enough that it is crucial to know they can exist, and that when they do, they encode important hidden information. In short, we not only demonstrate that DQC can be flexibly applied to datasets that present significantly different challenges, we also show how a simple analysis can be used to look for the needle in the haystack, determine what it is, and find what this means.
---
paper_title: A scalable inline cluster deduplication framework for big data protection

paper_content:

Cluster deduplication has become a widely deployed technology in data protection services for Big Data to satisfy the requirements of service level agreement (SLA). However, it remains a great challenge for cluster deduplication to strike a sensible tradeoff between the conflicting goals of scalable deduplication throughput and high duplicate elimination ratio in cluster systems with low-end individual secondary storage nodes. We propose Σ-Dedupe, a scalable inline cluster deduplication framework, as a middleware deployable in cloud data centers, to meet this challenge by exploiting data similarity and locality to optimize cluster deduplication in inter-node and intra-node scenarios, respectively. Governed by a similarity-based stateful data routing scheme, Σ-Dedupe assigns similar data to the same backup server at the super-chunk granularity using a handprinting technique to maintain high cluster-deduplication efficiency without cross-node deduplication, and balances the workload of servers from backup clients. Meanwhile, Σ-Dedupe builds a similarity index over the traditional locality-preserved caching design to alleviate the chunk index-lookup bottleneck in each node. Extensive evaluation of our Σ-Dedupe prototype against state-of-the-art schemes, driven by real-world datasets, demonstrates that Σ-Dedupe achieves a cluster-wide duplicate elimination ratio almost as high as the high-overhead and poorly scalable traditional stateful routing scheme but at an overhead only slightly higher than that of the scalable but low duplicate-elimination-ratio stateless routing approaches.
---
paper_title: BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables

paper_content:

The l1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix even under high-dimensional settings. However, it requires solving a difficult non-smooth log-determinant program with number of parameters scaling quadratically with the number of Gaussian variables. State-of-the-art methods thus do not scale to problems with more than 20,000 variables. In this paper, we develop an algorithm BIGQUIC, which can solve 1 million dimensional l1-regularized Gaussian MLE problems (which would thus have 1000 billion parameters) using a single machine, with bounded memory. In order to do so, we carefully exploit the underlying structure of the problem. Our innovations include a novel block-coordinate descent method with the blocks chosen via a clustering scheme to minimize repeated computations; and allowing for inexact computation of specific components. In spite of these modifications, we are able to theoretically analyze our procedure and show that BIGQUIC can achieve super-linear or even quadratic convergence rates.
---
paper_title: Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clustering

paper_content:

We develop and analyze a method to reduce the size of a very large set of data points in a high dimensional Euclidean space R d to a small set of weighted points such that the result of a predetermined data analysis task on the reduced set is approximately the same as that for the original point set. For example, computing the first k principal components of the reduced set will return approximately the first k principal components of the original set or computing the centers of a k-means clustering on the reduced set will return an approximation for the original set. Such a reduced set is also known as a coreset. The main new feature of our construction is that the cardinality of the reduced set is independent of the dimension d of the input space and that the sets are mergable. The latter property means that the union of two reduced sets is a reduced set for the union of the two original sets (this property has recently also been called composability, see Indyk et. al., PODS 2014). It allows us to turn our methods into streaming or distributed algorithms using standard approaches. For problems such as k-means and subspace approximation the coreset sizes are also independent of the number of input points. Our method is based on projecting the points on a low dimensional subspace and reducing the cardinality of the points inside this subspace using known methods. The proposed approach works for a wide range of data analysis techniques including k-means clustering, principal component analysis and subspace clustering. The main conceptual contribution is a new coreset definition that allows to charge costs that appear for every solution to an additive constant.
---
paper_title: Data mining with big data

paper_content:

Big Data concern large-volume, complex, growing data sets with multiple, autonomous sources. With the fast development of networking, data storage, and the data collection capacity, Big Data are now rapidly expanding in all science and engineering domains, including physical, biological and biomedical sciences. This paper presents a HACE theorem that characterizes the features of the Big Data revolution, and proposes a Big Data processing model, from the data mining perspective. This data-driven model involves demand-driven aggregation of information sources, mining and analysis, user interest modeling, and security and privacy considerations. We analyze the challenging issues in the data-driven model and also in the Big Data revolution.
---
paper_title: Big Data Deep Learning: Challenges and Perspectives

paper_content:

Deep learning is currently an extremely active research area in machine learning and pattern recognition society. It has gained huge successes in a broad area of applications such as speech recognition, computer vision, and natural language processing. With the sheer size of data available today, big data brings big opportunities and transformative potential for various sectors; on the other hand, it also presents unprecedented challenges to harnessing data and information. As the data keeps getting bigger, deep learning is coming to play a key role in providing big data predictive analytics solutions. In this paper, we provide a brief overview of deep learning, and highlight current research efforts and the challenges to big data, as well as the future trends.
---
paper_title: Temporal Analytics on Big Data for Web Advertising

paper_content:

"Big Data" in map-reduce (M-R) clusters is often fundamentally temporal in nature, as are many analytics tasks over such data. For instance, display advertising uses Behavioral Targeting (BT) to select ads for users based on prior searches, page views, etc. Previous work on BT has focused on techniques that scale well for offline data using M-R. However, this approach has limitations for BT-style applications that deal with temporal data: (1) many queries are temporal and not easily expressible in M-R, and moreover, the set-oriented nature of M-R front-ends such as SCOPE is not suitable for temporal processing, (2) as commercial systems mature, they may need to also directly analyze and react to real-time data feeds since a high turnaround time can result in missed opportunities, but it is difficult for current solutions to naturally also operate over real-time streams. Our contributions are twofold. First, we propose a novel framework called TiMR (pronounced timer), that combines a time-oriented data processing system with a M-R framework. Users write and submit analysis algorithms as temporal queries - these queries are succinct, scale-out-agnostic, and easy to write. They scale well on large-scale offline data using TiMR, and can work unmodified over real-time streams. We also propose new cost-based query fragmentation and temporal partitioning schemes for improving efficiency with TiMR. Second, we show the feasibility of this approach for BT, with new temporal algorithms that exploit new targeting opportunities. Experiments using real data from a commercial ad platform show that TiMR is very efficient and incurs orders-of-magnitude lower development effort. Our BT solution is easy and succinct, and performs up to several times better than current schemes in terms of memory, learning time, and click-through-rate/coverage.
---
paper_title: An Analytical Tool to Map Big Data to Networks with Reduced Topologies

paper_content:

The topological and dynamical properties of real-world networks have attracted extensive research from a variety of multi-disciplinary fields. They, in fact, model typically big datasets which pose interesting challenges, due to their intrinsic size and complex interactions, as well as the dependencies between their different sub-parts. Therefore, defining networks based on such properties, is unlikely to produce usable information due to their complexity and the data inconsistencies which they typically contain. In this paper, we discuss the evaluation of a method as part of ongoing research which aims to mine data to assess whether their associated networks exhibit properties comparable to well-known structures, namely scale-free, small world and random networks. For this, we will use a large dataset containing information on the seismologic activity recorded by the European-Mediterranean Seismological Centre. We will show that it provides an accurate, agile, and scalable tool to extract useful information. This further motivates our effort to produce a big data analytics tool which will focus on obtaining in-depth intelligence from both structured and unstructured big datasets. This will ultimately lead to a better understanding and prediction of the properties of the system(s) they model.
---
paper_title: Big data challenges and opportunities in high-throughput sequencing

paper_content:

The advent of high-throughput sequencing, coupled with advances in computational methods, has enabled genome-wide dissection of genetics, evolution, and disease, with nucleotide resolution. The discoveries derived from genomics promise benefits to basic research, biotechnology, and medicine; however, the speed and affordability of sequencing has resulted in a flood of “big data” in the life sciences. In addition, the current heterogeneity of sequencing platforms and diversity of applications complicate the development of tools for analysis, and this has slowed widespread adoption of the technology. Making sense of the data and delivering actionable insight requires improved computational infrastructure, new methods for interpreting the data, and unique collaborative approaches. Here we review the role of big data in genomics, its impact on the development of tools for collaborative analysis of genomes, and successes and ongoing challenges in coping with big data.
---
paper_title: Big data (lost) in the cloud

paper_content:

The big data era poses a critically difficult challenge and striking development opportunities to high performance computing (HPC). The major problem is an efficient transformation of the massive data of various types into valuable information and meaningful knowledge. Computationally-effective HPC is required in a fast-increasing number of data-intensive domains. With its special features of self-service and pay-as-you-use, cloud computing (CC) offers suitable abstractions to manage the complexity of the analysis of large data in various scientific and engineering domains. This paper surveys briefly the most recent developments on CC support for solving the big data problems. It presents a comprehensive critical analysis of the existing solutions and shows the further possible directions of the research in this domain including new generation multi-datacentre cloud architectures for the storage and management of the huge big data streams.
---
paper_title: Big data challenge: a data management perspective

paper_content:

There is a trend that, virtually everyone, ranging from big Web companies to traditional enterprisers to physical science researchers to social scientists, is either already experiencing or anticipating unprecedented growth in the amount of data available in their world, as well as new opportunities and great untapped value. This paper reviews big data challenges from a data management respective. In particular, we discuss big data diversity, big data reduction, big data integration and cleaning, big data indexing and query, and finally big data analysis and mining. Our survey gives a brief overview about big-data-oriented research and problems.
---
paper_title: The Emerging "Big Dimensionality"

paper_content:

The world continues to generate quintillion bytes of data daily, leading to the pressing needs for new efforts in dealing with the grand challenges brought by Big Data. Today, there is a growing consensus among the computational intelligence communities that data volume presents an immediate challenge pertaining to the scalability issue. However, when addressing volume in Big Data analytics, researchers in the data analytics community have largely taken a one-sided study of volume, which is the "Big Instance Size" factor of the data. The flip side of volume which is the dimensionality factor of Big Data, on the other hand, has received much lesser attention. This article thus represents an attempt to fill in this gap and places special focus on this relatively under-explored topic of "Big Dimensionality", wherein the explosion of features (variables) brings about new challenges to computational intelligence. We begin with an analysis on the origins of Big Dimensionality. The evolution of feature dimensionality in the last two decades is then studied using popular data repositories considered in the data analytics and computational intelligence research communities. Subsequently, the state-of-the-art feature selection schemes reported in the field of computational intelligence are reviewed to reveal the inadequacies of existing approaches in keeping pace with the emerging phenomenon of Big Dimensionality. Last but not least, the "curse and blessing of Big Dimensionality" are delineated and deliberated.
---
paper_title: Malicious URL filtering — A big data application

paper_content:

Malicious URLs have become a channel for Internet criminal activities such as drive-by-download, spamming and phishing. Applications for the detection of malicious URLs are accurate but slow (because they need to download the content or query some Internet host information). In this paper we present a novel lightweight filter based only on the URL string itself to use before existing processing methods. We run experiments on a large dataset and demonstrate a 75% reduction in workload size while retaining at least 90% of malicious URLs. Existing methods do not scale well with the hundreds of millions of URLs encountered every day as the problem is a heavily-imbalanced, large-scale binary classification problem. Our proposed method is able to handle nearly two million URLs in less than five minutes. We generate two filtering models by using lexical features and descriptive features, and then combine the filtering results. The on-line learning algorithms are applied here not only for dealing with large-scale data sets but also for fitting the very short lifetime characteristics of malicious URLs. Our filter can significantly reduce the volume of URL queries on which further analysis needs to be performed, saving both computing time and bandwidth used for content retrieval.
---
paper_title: Trends in big data analytics

paper_content:

Abstract One of the major applications of future generation parallel and distributed systems is in big-data analytics. Data repositories for such applications currently exceed exabytes and are rapidly increasing in size. Beyond their sheer magnitude, these datasets and associated applications’ considerations pose significant challenges for method and software development. Datasets are often distributed and their size and privacy considerations warrant distributed techniques. Data often resides on platforms with widely varying computational and network capabilities. Considerations of fault-tolerance, security, and access control are critical in many applications (Dean and Ghemawat, 2004; Apache hadoop). Analysis tasks often have hard deadlines, and data quality is a major concern in yet other applications. For most emerging applications, data-driven models and methods, capable of operating at scale, are as-yet unknown. Even when known methods can be scaled, validation of results is a major issue. Characteristics of hardware platforms and the software stack fundamentally impact data analytics. In this article, we provide an overview of the state-of-the-art and focus on emerging trends to highlight the hardware, software, and application landscape of big-data analytics.
---
paper_title: Where Chemical Sensors May Assist in Clinical Diagnosis Exploring “Big Data”

paper_content:

An overview is presented of the many opportunities and challenges for using “Big Data” concepts to treat data from chemical sensors. Issues discussed include the need to make data machine readable ...
---
paper_title: UniMiner: Towards a unified framework for data mining

paper_content:

Wearable devices and Smartphones generate huge data streams in pervasive and ubiquitous environments. Traditionally, big data systems collect all the data at a central data processing system (DPS). These data silos are further analyzed to generate approximated patterns for different application areas. This approach has one-sided utility (i.e. at big data processing end) but two main side-effects that lead towards user's dissatisfaction and extra computational costs. These effects are: 1) since all the data is being collected at central DPS, user privacy is compromised and 2) the collection of huge raw data streams, most of which could be irrelevant, at central systems needs more computational and storage resources hence increases the overall operational cost. Keeping in view these limitations, we are proposing a unified framework that balances between utility and cost of big data system with increased user satisfaction. We studied different data mining systems and proposed a new framework, named as UniMiner, to leverage data mining systems with wearable devices, smartphones, and cloud computing technologies. The gist of UniMiner is the scalability of data mining tasks from resource-constraint devices to collaborative and hybrid execution models. This scalable unified data mining approach distinguishes UniMiner from existing systems by enabling maximum data processing near data sources. Finally, we assessed the feasibility of mobile devices using six frequent pattern mining algorithms. The results show that mobile devices could be adopted as data mining platforms by tuning some additional parameters.
---
paper_title: Scalable data summarization on big data

paper_content:

Across different scientific domains, engineering disciplines, and application scenarios, increasingly, users have to deal with large-scale, diverse, feature-rich, and highresolution data sets that allow for data-intensive decision-making. The so-called big data challenge is making a profound transformation in computing. Big data not only refers to data sets that are large in size, but also covers data sets that are complex in structures, high dimensional, distributed, and heterogeneous. An effective frameworkwhen working with big data is through data summaries, such as different sampling methods, histograms, sketches and synopses, low-rank subspace approximation, dimensionality reduction techniques, etc. Instead of operating on complex and large raw data directly, these tools enable the execution of various data analytics tasks through appropriate and carefully constructed summaries, which improve their efficiency and scalability. Though some of these topics have been well studied in the past, the big data phenomena opens doors for interesting new research. These challenges include, but are not limited to, how to quantify the accuracy and efficiency trade-offwhen summarizing big data in massively parallel and distributed environments, how to summarize features in complex heterogeneous data, how to address IO and system issues in a summarization process, how to reduce communication cost when building a summary for a large data set stored in a cluster of commodity machines (such as a key-value store), how to dynamically maintain a summary in an incremental fashion under arbitrary arrivals of new data. As a result, answering the big data challenge through scalable data summarization is becoming of paramount importance.
---
paper_title: Can we analyze big data inside a DBMS?

paper_content:

Relational DBMSs remain the main data management technology, despite the big data analytics and no-SQL waves. On the other hand, for data analytics in a broad sense, there are plenty of non-DBMS tools including statistical languages, matrix packages, generic data mining programs and large-scale parallel systems, being the main technology for big data analytics. Such large-scale systems are mostly based on the Hadoop distributed file system and MapReduce. Thus it would seem a DBMS is not a good technology to analyze big data, going beyond SQL queries, acting just as a reliable and fast data repository. In this survey, we argue that is not the case, explaining important research that has enabled analytics on large databases inside a DBMS. However, we also argue DBMSs cannot compete with parallel systems like MapReduce to analyze web-scale text data. Therefore, each technology will keep influencing each other. We conclude with a proposal of long-term research issues, considering the "big data analytics" trend.
---
paper_title: Participatory Personal Data: An Emerging Research Challenge for the Information Sciences

paper_content:

Individuals can increasingly collect data about their habits, routines, and environment using ubiquitous technologies. Running specialized software, personal devices such as phones and tablets can capture and transmit users’ location, images, motion, and text input. The data collected by these devices are both personal (identifying of an individual) and participatory (accessible by that individual for aggregation, analysis, and sharing). Such participatory personal data provide a new area of inquiry for the information sciences. This article presents a review of literature from diverse fields, including information science, technology studies, surveillance studies, and participatory research traditions to explore how participatory personal data relate to existing personal data collections created by both research and surveillance. It applies three information perspectives—information policy, information access and equity, and data curation and preservation—to illustrate social impacts and concerns engendered by this new form of data collection. These perspectives suggest a set of research challenges for information science posed by participatory personal data. © 2012 Wiley Periodicals, Inc.
---
paper_title: Fog Computing: A Platform for Internet of Things and Analytics

paper_content:

Internet of Things (IoT) brings more than an explosive proliferation of endpoints. It is disruptive in several ways. In this chapter we examine those disruptions, and propose a hierarchical distributed architecture that extends from the edge of the network to the core nicknamed Fog Computing. In particular, we pay attention to a new dimension that IoT adds to Big Data and Analytics: a massively distributed number of sources at the edge.
---
paper_title: Energy-efficient data centers

paper_content:

Energy consumption of the Information and Communication Technology (ICT) sector has grown exponentially in recent years. A major component of the today’s ICT is constituted by the data centers which have experienced an unprecedented growth in their size and population, recently. The Internet giants like Google, IBM and Microsoft house large data centers for cloud computing and application hosting. Many studies, on energy consumption of data centers, point out to the need to evolve strategies for energy efficiency. Due to large-scale carbon dioxide (\(\mathrm{CO}_2\)) emissions, in the process of electricity production, the ICT facilities are indirectly responsible for considerable amounts of green house gas emissions. Heat generated by these densely populated data centers needs large cooling units to keep temperatures within the operational range. These cooling units, obviously, escalate the total energy consumption and have their own carbon footprint. In this survey, we discuss various aspects of the energy efficiency in data centers with the added emphasis on its motivation for data centers. In addition, we discuss various research ideas, industry adopted techniques and the issues that need our immediate attention in the context of energy efficiency in data centers.
---
paper_title: Machine learning for Big Data analytics in plants

paper_content:

Rapid advances in high-throughput genomic technology have enabled biology to enter the era of ‘Big Data' (large datasets). The plant science community not only needs to build its own Big-Data-compatible parallel computing and data management infrastructures, but also to seek novel analytical paradigms to extract information from the overwhelming amounts of data. Machine learning offers promising computational and analytical solutions for the integrative analysis of large, heterogeneous and unstructured datasets on the Big-Data scale, and is gradually gaining popularity in biology. This review introduces the basic concepts and procedures of machine-learning applications and envisages how machine learning could interface with Big Data technology to facilitate basic research and biotechnology in the plant sciences.
---
paper_title: Significance and Challenges of Big Data Research

paper_content:

In recent years, the rapid development of Internet, Internet of Things, and Cloud Computing have led to the explosive growth of data in almost every industry and business area. Big data has rapidly developed into a hot topic that attracts extensive attention from academia, industry, and governments around the world. In this position paper, we first briefly introduce the concept of big data, including its definition, features, and value. We then identify from different perspectives the significance and opportunities that big data brings to us. Next, we present representative big data initiatives all over the world. We describe the grand challenges (namely, data complexity, computational complexity, and system complexity), as well as possible solutions to address these challenges. Finally, we conclude the paper by presenting several suggestions on carrying out big data projects.
---
paper_title: Towards Network Reduction on Big Data

paper_content:

The increasing ease of data collection experience and the increasing availability of large data storage space lead to the existence of very large datasets that are commonly referred as "Big Data". Such data not only take over large amount of database storage, but also increase the difficulties for data analysis due to data diversity, which, also makes the datasets seemingly isolated with each other. In this paper, we present a solution to the problem that is to build up connections among the diverse datasets, based upon their similarities. Particularly, a concept of similarity graph along with a similarity graph generation algorithm were introduced. We then proposed a similarity graph reduction algorithm that reduces vertices of the graph for the purpose of graph simplification.
---
paper_title: Computing persistent features in big data: A distributed dimension reduction approach

paper_content:

Persistent homology has become one of the most popular tools used in topological data analysis for analyzing big data sets. In an effort to minimize the computational complexity of finding the persistent homology of a data set, we develop a simplicial collapse algorithm called the selective collapse. This algorithm works by representing the previously developed strong collapse as a forest and uses that forest data to improve the speed of both the strong collapse and of persistent homology. Finally, we demonstrate the savings in computational complexity using geometric random graphs.
---
paper_title: Computing Topological Persistence for Simplicial Maps

paper_content:

Algorithms for persistent homology are well-studied where homomorphisms are induced by inclusion maps. In this paper, we propose a practical algorithm for computing persistence under Z2 coefficients for a (monotone) sequence of general simplicial maps and show how these maps arise naturally in some applications of topological data analysis. ::: A simplicial map can be decomposed into a set of elementary inclusions and vertex collapses--two atomic operations that can be supported efficiently with the notion of simplex annotations for computing persistent homology. A consistent annotation through these atomic operations implies the maintenance of a consistent cohomology basis, hence a homology basis by duality. While the idea of maintaining a cohomology basis through an inclusion is not new, maintaining them through a vertex collapse is new, which constitutes an important atomic operation for simulating simplicial maps. Annotations support the vertex collapse in addition to the usual inclusion quite naturally. ::: Finally, we exhibit an application of this new tool in which we approximate the persistence diagram of a filtration of Rips complexes where vertex collapses are used to tame the blow-up in size.
---
paper_title: Reduced topologically real-world networks: a big-data approach

paper_content:

The topological and dynamical properties of real-world networks have attracted extensive research from a variety of multi-disciplinary fields. They, in fact, model typically big datasets which pose interesting challenges, due to their intrinsic size and complex interactions, as well as the dependencies between their different sub-parts. Therefore, defining networks based on such properties, is unlikely to produce usable information due to their complexity and the data inconsistencies which they typically contain. In this paper, the authors discuss the evaluation of a method as part of ongoing research which aims to mine data to assess whether their associated networks exhibit properties comparable to well-known structures, namely scale-free, small world and random networks. For this, they will use a large dataset containing information on the seismologic activity recorded by the European-Mediterranean Seismological Centre. The authors will show that it provides an accurate, agile, and scalable tool to extract useful information. This further motivates their effort to produce a big data analytics tool which will focus on obtaining in-depth intelligence from both structured and unstructured big datasets. This will ultimately lead to a better understanding and prediction of the properties of the system(s) they model.
---
paper_title: An influence assessment method based on co-occurrence for topologically reduced big data sets

paper_content:

The extraction of meaningful, accurate, and relevant information is at the core of Big Data research. Furthermore, the ability to obtain an insight is essential in any decision-making process, even though the diverse and complex nature of big data sets raises a multitude of challenges. In this paper, we propose a novel method to address the automated assessment of influence among concepts in big data sets. This is carried out by investigating their mutual co-occurrence, which is determined via topologically reducing the corresponding network. The main motivation is to provide a toolbox to classify and analyse influence properties, which can be used to investigate their dynamical and statistical behaviour, potentially leading to a better understanding and prediction of the properties of the system(s) they model. An evaluation was carried out on two real-world data sets, which were analysed to test the capabilities of our system. The results show the potential of our approach, indicating both accuracy and efficiency.
---
paper_title: FlexAnalytics: A flexible data analytics framework for big data applications with I/O performance improvement

paper_content:

Increasingly larger scale applications are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. Instead of moving data from its source to the output storage, in-situ analytics processes output data while simulations are running. However, in-situ data analysis incurs much more computing resource contentions with simulations. Such contentions severely damage the performance of simulation on HPE. Since different data processing strategies have different impact on performance and cost, there is a consequent need for flexibility in the location of data analytics. In this paper, we explore and analyze several potential data-analytics placement strategies along the I/O path. To find out the best strategy to reduce data movement in given situation, we propose a flexible data analytics (FlexAnalytics) framework in this paper. Based on this framework, a FlexAnalytics prototype system is developed for analytics placement. FlexAnalytics system enhances the scalability and flexibility of current I/O stack on HEC platforms and is useful for data pre-processing, runtime data analysis and visualization, as well as for large-scale data transfer. Two use cases – scientific data compression and remote visualization – have been applied in the study to verify the performance of FlexAnalytics. Experimental results demonstrate that FlexAnalytics framework increases data transition bandwidth and improves the application end-to-end transfer performance.
---
paper_title: Interactive remote large-scale data visualization via prioritized multi-resolution streaming

paper_content:

The simulations that run on petascale and future exascale supercomputers pose a difficult challenge for scientists to visualize and analyze their results remotely. They are limited in their ability to interactively visualize their data mainly due to limited network bandwidth associated with sending and reading large data at a distance. To tackle this issue, we provide a generalized distance visualization architecture for large remote data that aims to provide interactive analysis. We achieve this through a prioritized, multi-resolution, streaming architecture. Since the original data size is several orders of magnitude greater than the display and network technologies, we stream downsampled versions of representation data over time to complete a visualization using fast local rendering. This technique provides the necessary interactivity and full-resolution results dynamically on demand while maintaining a full-featured visualization framework.
---
paper_title: The Anamorphic Stretch Transform: Putting the Squeeze on “Big Data”

paper_content:

Coping with a deluge of digital information will require more efficient ways to capture, ::: sample and store data. One new approach works by selectively “warping” the data to provide ::: better resolution of the fine details - while still reducing total data size.
---
paper_title: Big Data Reduction and Optimization in Sensor Monitoring Network

paper_content:

Wireless sensor networks (WSNs) are increasingly being utilized to monitor the structural health of the underground subway tunnels, showing many promising advantages over traditional monitoring schemes. Meanwhile, with the increase of the network size, the system is incapable of dealing with big data to ensure efficient data communication, transmission, and storage. Being considered as a feasible solution to these issues, data compression can reduce the volume of data travelling between sensor nodes. In this paper, an optimization algorithm based on the spatial and temporal data compression is proposed to cope with these issues appearing in WSNs in the underground tunnel environment. The spatial and temporal correlation functions are introduced for the data compression and data recovery. It is verified that the proposed algorithm is applicable to WSNs in the underground tunnel.
---
paper_title: Large-scale electrophysiology: acquisition, compression, encryption, and storage of big data.

paper_content:

Abstract The use of large-scale electrophysiology to obtain high spatiotemporal resolution brain recordings (>100 channels) capable of probing the range of neural activity from local field potential oscillations to single-neuron action potentials presents new challenges for data acquisition, storage, and analysis. Our group is currently performing continuous, long-term electrophysiological recordings in human subjects undergoing evaluation for epilepsy surgery using hybrid intracranial electrodes composed of up to 320 micro- and clinical macroelectrode arrays. DC-capable amplifiers, sampling at 32 kHz per channel with 18-bits of A/D resolution are capable of resolving extracellular voltages spanning single-neuron action potentials, high frequency oscillations, and high amplitude ultra-slow activity, but this approach generates 3 terabytes of data per day (at 4 bytes per sample) using current data formats. Data compression can provide several practical benefits, but only if data can be compressed and appended to files in real-time in a format that allows random access to data segments of varying size. Here we describe a state-of-the-art, scalable, electrophysiology platform designed for acquisition, compression, encryption, and storage of large-scale data. Data are stored in a file format that incorporates lossless data compression using range-encoded differences, a 32-bit cyclically redundant checksum to ensure data integrity, and 128-bit encryption for protection of patient information.
---
paper_title: Improving I/O Performance with Adaptive Data Compression for Big Data Applications

paper_content:

Increasingly larger scale simulations are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. As a solution, in-situ analytics processes output data while simulations are running and before placing data on disk. Data movement between simulation and analytics, however, incurs overheads of in-situ analytics at scale. This paper tries to answer the following question: can we use compression technology to reduce the data movement cost and improve the performance of in-situ analytics for peta-scale applications? In particular, we explore when, where, how to use the compression techniques to reduce data movement cost between simulation and analytics. To find out the best algorithm and place to compress data in given situation, we introduce an adaptive data compression algorithm in this paper. The adaptive compression service is developed and analyzed for the in-situ analytics middleware. Experimental results demonstrate that compression service increases data transition bandwidth and improve the application End-to-End transfer performance.
---
paper_title: Compressing the incompressible with ISABELA: in-situ reduction of spatio-temporal data

paper_content:

Modern large-scale scientific simulations running on HPC systems generate data in the order of terabytes during a single run. To lessen the I/O load during a simulation run, scientists are forced to capture data infrequently, thereby making data collection an inherently lossy process. Yet, lossless compression techniques are hardly suitable for scientific data due to its inherently random nature; for the applications used here, they offer less than 10% compression rate. They also impose significant overhead during decompression, making them unsuitable for data analysis and visualization that require repeated data access. ::: ::: To address this problem, we propose an effective method for In-situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data that is widely regarded as effectively incompressible. With ISABELA, we apply a preconditioner to seemingly random and noisy data along spatial resolution to achieve an accurate fitting model that guarantees a ≥ 0.99 correlation with the original data. We further take advantage of temporal patterns in scientific data to compress data by ≈ 85%, while introducing only a negligible overhead on simulations in terms of runtime. ISABELA significantly outperforms existing lossy compression methods, such as Wavelet compression. Moreover, besides being a communication-free and scalable compression technique, ISABELA is an inherently local decompression method, namely it does not decode the entire data, making it attractive for random access.
---
paper_title: Statistical wavelet-based anomaly detection in big data with compressive sensing

paper_content:

Anomaly detection in big data is a key problem in the big data analytics domain. In this paper, the definitions of anomaly detection and big data were presented. Due to the sampling and storage burden and the inadequacy of privacy protection of anomaly detection based on uncompressed data, compressive sensing theory was introduced and used in the anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The proposed anomaly detection technique was used for through-wall human detection to demonstrate the effectiveness. The experiments for detecting humans behind a brick wall and gypsum based on ultra-wideband radar signal were carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of a human being through compressed signals and uncompressed data.
---
paper_title: A Resource Efficient Big Data Analysis Method for the Social Sciences: The Case of Global IP Activity☆

paper_content:

Abstract This paper presents a novel and efficient way of analysing big datasets used in social science research. We provide and demonstrate a way to deal with such datasets without the need for high performance distributed computational facilities. Using an Internet census dataset and with the help of freely available tools and programming libraries, we visualize global IP activity in a spatial and time dimension. We observe a considerable reduction in storage size of our dataset coupled with a faster processing time.
---
paper_title: A spatiotemporal compression based approach for efficient big data processing on Cloud

paper_content:

It is well known that processing big graph data can be costly on Cloud. Processing big graph data introduces complex and multiple iterations that raise challenges such as parallel memory bottlenecks, deadlocks, and inefficiency. To tackle the challenges, we propose a novel technique for effectively processing big graph data on Cloud. Specifically, the big data will be compressed with its spatiotemporal features on Cloud. By exploring spatial data correlation, we partition a graph data set into clusters. In a cluster, the workload can be shared by the inference based on time series similarity. By exploiting temporal correlation, in each time series or a single graph edge, temporal data compression is conducted. A novel data driven scheduling is also developed for data processing optimisation. The experiment results demonstrate that the spatiotemporal compression and scheduling achieve significant performance gains in terms of data size and data fidelity loss.
---
paper_title: Privacy-Preserving Distributed Movement Data Aggregation

paper_content:

We propose a novel approach to privacy-preserving analytical processing within a distributed setting, and tackle the problem of obtaining aggregated information about vehicle traffic in a city from movement data collected by individual vehicles and shipped to a central server. Movement data are sensitive because people’s whereabouts have the potential to reveal intimate personal traits, such as religious or sexual preferences, and may allow re-identification of individuals in a database. We provide a privacy-preserving framework for movement data aggregation based on trajectory generalization in a distributed environment. The proposed solution, based on the differential privacy model and on sketching techniques for efficient data compression, provides a formal data protection safeguard. Using real-life data, we demonstrate the effectiveness of our approach also in terms of data utility preserved by the data transformation.
---
paper_title: Tradeoffs in scalable data routing for deduplication clusters

paper_content:

As data have been growing rapidly in data centers, deduplication storage systems continuously face challenges in providing the corresponding throughputs and capacities necessary to move backup data within backup and recovery window times. One approach is to build a cluster deduplication storage system with multiple deduplication storage system nodes. The goal is to achieve scalable throughput and capacity using extremely high-throughput (e.g. 1.5 GB/s) nodes, with a minimal loss of compression ratio. The key technical issue is to route data intelligently at an appropriate granularity. ::: ::: We present a cluster-based deduplication system that can deduplicate with high throughput, support deduplication ratios comparable to that of a single system, and maintain a low variation in the storage utilization of individual nodes. In experiments with dozens of nodes, we examine tradeoffs between stateless data routing approaches with low overhead and stateful approaches that have higher overhead but avoid imbalances that can adversely affect deduplication effectiveness for some datasets in large clusters. The stateless approach has been deployed in a two-node commercial system that achieves 3 GB/s for multi-stream deduplication throughput and currently scales to 5.6 PB of storage (assuming 20X total compression).
---
paper_title: Characterizing the efficiency of data deduplication for big data storage management

paper_content:

The demand for data storage and processing is increasing at a rapid speed in the big data era. Such a tremendous amount of data pushes the limit on storage capacity and on the storage network. A significant portion of the dataset in big data workloads is redundant. As a result, deduplication technology, which removes replicas, becomes an attractive solution to save disk space and traffic in a big data environment. However, the overhead of extra CPU computation (hash indexing) and IO latency introduced by deduplication should be considered. Therefore, the net effect of using deduplication for big data workloads needs to be examined. To this end, we characterize the redundancy of typical big data workloads to justify the need for deduplication. We analyze and characterize the performance and energy impact brought by deduplication under various big data environments. In our experiments, we identify three sources of redundancy in big data workloads: 1) deploying more nodes, 2) expanding the dataset, and 3) using replication mechanisms. We elaborate on the advantages and disadvantages of different deduplication layers, locations, and granularities. In addition, we uncover the relation between energy overhead and the degree of redundancy. Furthermore, we investigate the deduplication efficiency in an SSD environment for big data workloads.
---
paper_title: Exploiting in-network processing for big data management

paper_content:

Data processing systems face the task of efficiently storing and processing data at petabyte scale, with the amount set to increase in the future. To meet such a requirement, highly scalable, shared-nothing systems, e.g. Google's BigTable [6] or Facebook's Cassandra [14], are built to partition data and process it in parallel on distributed nodes in a cluster. This allows the handling of data at scale but introduces new challenges due to the distribution of data. Running queries involves a high network overhead because data has to be exchanged between cluster nodes and hence, the network becomes a critical part of the system. To avoid the network bottleneck, it is essential for distributed data processing systems (DDPS) to be aware of the network rather than treating it as a black box. We propose in-network processing as a way of achieving network-awareness to decrease bandwidth usage by custom routing, redundancy elimination, and on-path data reduction. Thereby, we can increase the query throughput of a DDPS. The challenges of an in-network processing system range from design issues, such as performance and transparency, to the integration with query optimisation and deployment in data centres. We formulate these challenges as possible research directions and provide a prototype implementation. Our preliminary results suggest that we can significantly improve query throughput in a DDPS by performing partial data reduction within the network.
---
paper_title: Silo: A similarity-locality based near-exact deduplication scheme with low ram overhead and high throughput

paper_content:

Data Deduplication is becoming increasingly popular in storage systems as a space-efficient approach to data backup and archiving. Most existing state-of-the-art deduplication methods are either locality based or similarity based, which, according to our analysis, do not work adequately in many situations. While the former produces poor deduplication throughput when there is little or no locality in datasets, the latter can fail to identify and thus remove significant amounts of redundant data when there is a lack of similarity among files. In this paper, we present SiLo, a near-exact deduplication system that effectively and complementarily exploits similarity and locality to achieve high duplicate elimination and throughput at extremely low RAM overheads. The main idea behind SiLo is to expose and exploit more similarity by grouping strongly correlated small files into a segment and segmenting large files, and to leverage locality in the backup stream by grouping contiguous segments into blocks to capture similar and duplicate data missed by the probabilistic similarity detection. By judiciously enhancing similarity through the exploitation of locality and vice versa, the SiLo approach is able to significantly reduce RAM usage for index-lookup and maintain a very high deduplication throughput. Our experimental evaluation of SiLo based on real-world datasets shows that the SiLo system consistently and significantly outperforms two existing state-of-the-art system, one based on similarity and the other based on locality, under various workload conditions.
---
paper_title: COUPON: A Cooperative Framework for Building Sensing Maps in Mobile Opportunistic Networks

paper_content:

Human-carried or vehicle-mounted sensors can be exploited to collect data ubiquitously for building various sensing maps. Most of existing mobile sensing applications consider users reporting and accessing sensing data through the Internet. However, this approach cannot be applied in the scenarios with poor network coverage or expensive network access. Existing data forwarding schemes for mobile opportunistic networks are not sufficient for sensing applications as spatial-temporal correlation among sensory data has not been explored. In order to build sensing maps satisfying specific sensing quality with low delay and energy consumption, we design COUPON, a novel cooperative sensing and data forwarding framework. We first notice that cooperative sensing scheme can eliminate sampling redundancy and hence save energy. Then we design two cooperative forwarding schemes by leveraging data fusion: Epidemic Routing with Fusion (ERF) and Binary Spray-and-Wait with Fusion (BSWF). Different from previous work assuming that all packets are propagated independently, we consider that packets are spatial-temporal correlated in the forwarding process, and derive the dissemination law of correlated packets. Both the theoretic analysis and simulation results show that our cooperative forwarding schemes can achieve better tradeoff between delivery delay and transmission overhead. We also evaluate our proposed framework and schemes with real mobile traces. Extensive simulations demonstrate that the cooperative sensing scheme can reduce the number of samplings by 93 percent compared with the non-cooperative scheme; ERF can reduce the transmission overhead by 78 percent compared with Epidemic Routing (ER); BSWF can increase the delivery ratio by 16 percent, and reduce the delivery delay and transmission overhead by 5 and 32 percent respectively, compared with Binary Spray-and-Wait (BSW).
---
paper_title: A scalable inline cluster deduplication framework for big data protection

paper_content:

Cluster deduplication has become a widely deployed technology in data protection services for Big Data to satisfy the requirements of service level agreement (SLA). However, it remains a great challenge for cluster deduplication to strike a sensible tradeoff between the conflicting goals of scalable deduplication throughput and high duplicate elimination ratio in cluster systems with low-end individual secondary storage nodes. We propose Σ-Dedupe, a scalable inline cluster deduplication framework, as a middleware deployable in cloud data centers, to meet this challenge by exploiting data similarity and locality to optimize cluster deduplication in inter-node and intra-node scenarios, respectively. Governed by a similarity-based stateful data routing scheme, Σ-Dedupe assigns similar data to the same backup server at the super-chunk granularity using a handprinting technique to maintain high cluster-deduplication efficiency without cross-node deduplication, and balances the workload of servers from backup clients. Meanwhile, Σ-Dedupe builds a similarity index over the traditional locality-preserved caching design to alleviate the chunk index-lookup bottleneck in each node. Extensive evaluation of our Σ-Dedupe prototype against state-of-the-art schemes, driven by real-world datasets, demonstrates that Σ-Dedupe achieves a cluster-wide duplicate elimination ratio almost as high as the high-overhead and poorly scalable traditional stateful routing scheme but at an overhead only slightly higher than that of the scalable but low duplicate-elimination-ratio stateless routing approaches.
---
paper_title: Content-based document routing and index partitioning for scalable similarity-based searches in a large corpus

paper_content:

We present a document routing and index partitioning scheme for scalable similarity-based search of documents in a large corpus. We consider the case when similarity-based search is performed by finding documents that have features in common with the query document. While it is possible to store all the features of all the documents in one index, this suffers from obvious scalability problems. Our approach is to partition the feature index into multiple smaller partitions that can be hosted on separate servers, enabling scalable and parallel search execution. When a document is ingested into the repository, a small number of partitions are chosen to store the features of the document. To perform similarity-based search, also, only a small number of partitions are queried. Our approach is stateless and incremental. The decision as to which partitions the features of the document should be routed to (for storing at ingestion time, and for similarity based search at query time) is solely based on the features of the document. Our approach scales very well. We show that executing similarity-based searches over such a partitioned search space has minimal impact on the precision and recall of search results, even though every search consults less than 3% of the total number of partitions.
---
paper_title: Big 3D spatial data processing using cloud computing environment

paper_content:

Lately, acquiring a large quantity of three-dimensional (3-D) spatial data particularly topographic information has become commonplace with the advent of new technology such as laser scanner or light detection and ranging (LiDAR) and techniques. Though both in the USA and around the globe, the pace of massive 3-D spatial data collection is accelerating, the provision of affordable technology for dealing with issues such as processing, management, archival, dissemination, and analysis of the huge data volumes has lagged behind. Single computers and generic high-end computing are not sufficient to process this massive data and researches started to explore other computing environments. Recently cloud computing environment showed very promising solutions due to availability and affordability. The main goal of this paper is to develop a web-based LiDAR data processing framework called "Cloud Computing-based LiDAR Processing System (CLiPS)" to process massive LiDAR data using cloud computing environment. The CLiPS framework implementation was done using ESRI's ArcGIS server, Amazon Elastic Compute Cloud (Amazon EC2), and several open source spatial tools. Some of the applications developed in this project include: 1) preprocessing tools for LiDAR data, 2) generation of large area Digital Elevation Model (DEMs) on the cloud environment, and 3) user-driven DEM derived products. We have used three different terrain types, LiDAR tile sizes, and EC2 instant types (large, Xlarge, and double Xlarge) to test for time and cost comparisons. Undulating terrain data took more time than other two terrain types used in this study and overall cost for the entire project was less than $100.
---
paper_title: Confidence-weighted linear classification

paper_content:

We introduce confidence-weighted linear classifiers, which add parameter confidence information to linear classifiers. Online learners in this setting update both classifier parameters and the estimate of their confidence. The particular online algorithms we study here maintain a Gaussian distribution over parameter vectors and update the mean and covariance of the distribution with each instance. Empirical evaluation on a range of NLP tasks show that our algorithm improves over other state of the art online and batch methods, learns faster in the online setting, and lends itself to better classifier combination after parallel training.
---
paper_title: Online Passive-Aggressive Algorithms

paper_content:

We present a unified view for online classification, regression, and uni-class problems. This view leads to a single algorithmic framework for the three problems. We prove worst case loss bounds for various algorithms for both the realizable case and the non-realizable case. A conversion of our main online algorithm to the setting of batch learning is also discussed. The end result is new algorithms and accompanying loss bounds for the hinge-loss.
---
paper_title: Big data (lost) in the cloud

paper_content:

The big data era poses a critically difficult challenge and striking development opportunities to high performance computing (HPC). The major problem is an efficient transformation of the massive data of various types into valuable information and meaningful knowledge. Computationally-effective HPC is required in a fast-increasing number of data-intensive domains. With its special features of self-service and pay-as-you-use, cloud computing (CC) offers suitable abstractions to manage the complexity of the analysis of large data in various scientific and engineering domains. This paper surveys briefly the most recent developments on CC support for solving the big data problems. It presents a comprehensive critical analysis of the existing solutions and shows the further possible directions of the research in this domain including new generation multi-datacentre cloud architectures for the storage and management of the huge big data streams.
---
paper_title: Increasing big data front end processing efficiency via locality sensitive Bloom filter for elderly healthcare

paper_content:

In support of the increasing number of elderly population, wearable sensors and portable mobile devices capable of monitoring, recording, reporting and alerting are envisaged to enable them an independent lifestyle without relying on intrusive care programmes. However, the big data readings generated from the sensors are characterized as multidimensional, dynamic and non-linear with weak correlation with observable human behaviors and health conditions which challenges the information transmission, storing and processing. This paper proposes to use Locality Sensitive Bloom Filter to increase the Instance Based Learning efficiency for the front end sensor data pre-processing so that only relevant and meaningful information will be sent out for further processing aiming to relieve the burden of the above big data challenges. The approach is proven to optimize and enhance a popular instance-based learning method benefits from its faster speed, less space requirements and is adequate for the application.
---
paper_title: Near Real-Time Processing of Proteomics Data Using Hadoop

paper_content:

Abstract This article presents a near real-time processing solution using MapReduce and Hadoop. The solution is aimed at some of the data management and processing challenges facing the life sciences community. Research into genes and their product proteins generates huge volumes of data that must be extensively preprocessed before any biological insight can be gained. In order to carry out this processing in a timely manner, we have investigated the use of techniques from the big data field. These are applied specifically to process data resulting from mass spectrometers in the course of proteomic experiments. Here we present methods of handling the raw data in Hadoop, and then we investigate a process for preprocessing the data using Java code and the MapReduce framework to identify 2D and 3D peaks.
---
paper_title: Malicious URL filtering — A big data application

paper_content:

Malicious URLs have become a channel for Internet criminal activities such as drive-by-download, spamming and phishing. Applications for the detection of malicious URLs are accurate but slow (because they need to download the content or query some Internet host information). In this paper we present a novel lightweight filter based only on the URL string itself to use before existing processing methods. We run experiments on a large dataset and demonstrate a 75% reduction in workload size while retaining at least 90% of malicious URLs. Existing methods do not scale well with the hundreds of millions of URLs encountered every day as the problem is a heavily-imbalanced, large-scale binary classification problem. Our proposed method is able to handle nearly two million URLs in less than five minutes. We generate two filtering models by using lexical features and descriptive features, and then combine the filtering results. The on-line learning algorithms are applied here not only for dealing with large-scale data sets but also for fitting the very short lifetime characteristics of malicious URLs. Our filter can significantly reduce the volume of URL queries on which further analysis needs to be performed, saving both computing time and bandwidth used for content retrieval.
---
paper_title: Challenges of Big Data Analysis

paper_content:

Big Data bring new opportunities to modern society and challenges to data scientists. On the one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This paper gives overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasize on the viability of the sparsest solution in high-confidence set and point out that exogenous assumptions in most statistical methods for Big Data cannot be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.
---
paper_title: Analyzing Big Data with Dynamic Quantum Clustering

paper_content:

How does one search for a needle in a multi-dimensional haystack without knowing what a needle is and without knowing if there is one in the haystack? This kind of problem requires a paradigm shift - away from hypothesis driven searches of the data - towards a methodology that lets the data speak for itself. Dynamic Quantum Clustering (DQC) is such a methodology. DQC is a powerful visual method that works with big, high-dimensional data. It exploits variations of the density of the data (in feature space) and unearths subsets of the data that exhibit correlations among all the measured variables. The outcome of a DQC analysis is a movie that shows how and why sets of data-points are eventually classified as members of simple clusters or as members of - what we call - extended structures. This allows DQC to be successfully used in a non-conventional exploratory mode where one searches data for unexpected information without the need to model the data. We show how this works for big, complex, real-world datasets that come from five distinct fields: i.e., x-ray nano-chemistry, condensed matter, biology, seismology and finance. These studies show how DQC excels at uncovering unexpected, small - but meaningful - subsets of the data that contain important information. We also establish an important new result: namely, that big, complex datasets often contain interesting structures that will be missed by many conventional clustering techniques. Experience shows that these structures appear frequently enough that it is crucial to know they can exist, and that when they do, they encode important hidden information. In short, we not only demonstrate that DQC can be flexibly applied to datasets that present significantly different challenges, we also show how a simple analysis can be used to look for the needle in the haystack, determine what it is, and find what this means.
---
paper_title: BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables

paper_content:

The l1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix even under high-dimensional settings. However, it requires solving a difficult non-smooth log-determinant program with number of parameters scaling quadratically with the number of Gaussian variables. State-of-the-art methods thus do not scale to problems with more than 20,000 variables. In this paper, we develop an algorithm BIGQUIC, which can solve 1 million dimensional l1-regularized Gaussian MLE problems (which would thus have 1000 billion parameters) using a single machine, with bounded memory. In order to do so, we carefully exploit the underlying structure of the problem. Our innovations include a novel block-coordinate descent method with the blocks chosen via a clustering scheme to minimize repeated computations; and allowing for inexact computation of specific components. In spite of these modifications, we are able to theoretically analyze our procedure and show that BIGQUIC can achieve super-linear or even quadratic convergence rates.
---
paper_title: Mammoth Data in the Cloud: Clustering Social Images

paper_content:

Social image datasets have grown to dramatic size with images classified in vector spaces with high dimension (512-2048) and with potentially billions of images and corresponding classification vectors. We study the challenging problem of clustering such sets into millions of clusters using Iterative MapReduce. We introduce a new Kmeans algorithm in the Map phase which can tackle the challenge of large cluster and dimension size. Further we stress that the necessary parallelism of such data intensive problems are dominated by particular collective (reduction) operations which are common to MPI and MapReduce and study different collective implementations, which enable cloud-HPC cluster interoperability. Extensive performance results are presented.
---
paper_title: Sparse inverse covariance estimation with the graphical lasso

paper_content:

SUMMARY We consider the problem of estimating sparse graphs by a lasso penalty applied to the inverse covariance matrix. Using a coordinate descent procedure for the lasso, we develop a simple algorithm—the graphical lasso—that is remarkably fast: It solves a 1000-node problem (∼500 000 parameters) in at most a minute and is 30–4000 times faster than competing methods. It also provides a conceptual link between the exact problem and the approximation suggested by Meinshausen and B¨ uhlmann (2006). We illustrate the method on some cell-signaling data from proteomics.
---
paper_title: Sparse Inverse Covariance Selection via Alternating Linearization Methods

paper_content:

Gaussian graphical models are of great interest in statistical learning. Because the conditional independencies between different nodes correspond to zero entries in the inverse covariance matrix of the Gaussian distribution, one can learn the structure of the graph by estimating a sparse inverse covariance matrix from sample data, by solving a convex maximum likelihood problem with an $\ell_1$-regularization term. In this paper, we propose a first-order method based on an alternating linearization technique that exploits the problem's special structure; in particular, the subproblems solved in each iteration have closed-form solutions. Moreover, our algorithm obtains an $\epsilon$-optimal solution in $O(1/\epsilon)$ iterations. Numerical experiments on both synthetic and real data from gene association networks show that a practical version of this algorithm outperforms other competitive algorithms.
---
paper_title: Dimensionality reduction of medical big data using neural-fuzzy classifier

paper_content:

Massive and complex data are generated every day in many fields. Complex data refer to data sets that are so large that conventional database management and data analysis tools are insufficient to deal with them. Managing and analysis of medical big data involve many different issues regarding their structure, storage and analysis. In this paper, linguistic hedges neuro-fuzzy classifier with selected features (LHNFCSF) is presented for dimensionality reduction, feature selection and classification. Four real-world data sets are provided to demonstrate the performance of the proposed neuro-fuzzy classifier. The new classifier is compared with the other classifiers for different classification problems. The results indicated that applying LHNFCSF not only reduces the dimensions of the problem, but also improves classification performance by discarding redundant, noise-corrupted, or unimportant features. The results strongly suggest that the proposed method not only help reducing the dimensionality of large data sets but also can speed up the computation time of a learning algorithm and simplify the classification tasks.
---
paper_title: Online feature selection for mining big data

paper_content:

Most studies of online learning require accessing all the attributes/features of training instances. Such a classical setting is not always appropriate for real-world applications when data instances are of high dimensionality or the access to it is expensive to acquire the full set of attributes/features. To address this limitation, we investigate the problem of Online Feature Selection (OFS) in which the online learner is only allowed to maintain a classifier involved a small and fixed number of features. The key challenge of Online Feature Selection is how to make accurate prediction using a small and fixed number of active features. This is in contrast to the classical setup of online learning where all the features are active and can be used for prediction. We address this challenge by studying sparsity regularization and truncation techniques. Specifically, we present an effective algorithm to solve the problem, give the theoretical analysis, and evaluate the empirical performance of the proposed algorithms for online feature selection on several public datasets. We also demonstrate the application of our online feature selection technique to tackle real-world problems of big data mining, which is significantly more scalable than some well-known batch feature selection algorithms. The encouraging results of our experiments validate the efficacy and efficiency of the proposed techniques for large-scale applications.
---
paper_title: Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis

paper_content:

Higher-order tensors and their decompositions are abundantly present in domains such as signal processing (e.g., higher-order statistics [1] and sensor array processing [2]), scientific computing (e.g., discretized multivariate functions [3]?[6]), and quantum information theory (e.g., representation of quantum many-body states [7]). In many applications, the possibly huge tensors can be approximated well by compact multilinear models or decompositions. Tensor decompositions are more versatile tools than the linear models resulting from traditional matrix approaches. Compared to matrices, tensors have at least one extra dimension. The number of elements in a tensor increases exponentially with the number of dimensions, and so do the computational and memory requirements. The exponential dependency (and the problems that are caused by it) is called the curse of dimensionality. The curse limits the order of the tensors that can be handled. Even for a modest order, tensor problems are often large scale. Large tensors can be handled, and the curse can be alleviated or even removed by using a decomposition that represents the tensor instead of using the tensor itself. However, most decomposition algorithms require full tensors, which renders these algorithms infeasible for many data sets. If a tensor can be represented by a decomposition, this hypothesized structure can be exploited by using compressed sensing (CS) methods working on incomplete tensors, i.e., tensors with only a few known elements.
---
paper_title: Incremental partial least squares analysis of big streaming data

paper_content:

Incremental feature extraction is effective for facilitating the analysis of large-scale streaming data. However, most current incremental feature extraction methods are not suitable for processing streaming data with high feature dimensions because only a few methods have low time complexity, which is linear with both the number of samples and features. In addition, feature extraction methods need to improve the performance of further classification. Therefore, incremental feature extraction methods need to be more efficient and effective. Partial least squares (PLS) is known to be an effective dimension reduction technique for classification. However, the application of PLS to streaming data is still an open problem. In this study, we propose a highly efficient and powerful dimension reduction algorithm called incremental PLS (IPLS), which comprises a two-stage extraction process. In the first stage, the PLS target function is adapted so it is incremental by updating the historical mean to extract the leading projection direction. In the second stage, the other projection directions are calculated based on the equivalence between the PLS vectors and the Krylov sequence. We compared the performance of IPLS with other state-of-the-art incremental feature extraction methods such as incremental principal components analysis, incremental maximum margin criterion, and incremental inter-class scatter using real streaming datasets. Our empirical results showed that IPLS performed better than other methods in terms of its efficiency and further classification accuracy.
---
paper_title: Rational Krylov sequence methods for eigenvalue computation

paper_content:

Abstract Algorithms to solve large sparse eigenvalue problems are considered. A new class of algorithms which is based on rational functions of the matrix is described. The Lanczos method, the Arnoldi method, the spectral transformation Lanczos method, and Rayleigh quotient iteration all are special cases, but there are also new algorithms which correspond to rational functions with several poles. In the simplest case a basis of a rational Krylov subspace is found in which the matrix eigenvalue problem is formulated as a linear matrix pencil with a pair of Hessenberg matrices.
---
paper_title: Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clustering

paper_content:

We develop and analyze a method to reduce the size of a very large set of data points in a high dimensional Euclidean space R d to a small set of weighted points such that the result of a predetermined data analysis task on the reduced set is approximately the same as that for the original point set. For example, computing the first k principal components of the reduced set will return approximately the first k principal components of the original set or computing the centers of a k-means clustering on the reduced set will return an approximation for the original set. Such a reduced set is also known as a coreset. The main new feature of our construction is that the cardinality of the reduced set is independent of the dimension d of the input space and that the sets are mergable. The latter property means that the union of two reduced sets is a reduced set for the union of the two original sets (this property has recently also been called composability, see Indyk et. al., PODS 2014). It allows us to turn our methods into streaming or distributed algorithms using standard approaches. For problems such as k-means and subspace approximation the coreset sizes are also independent of the number of input points. Our method is based on projecting the points on a low dimensional subspace and reducing the cardinality of the points inside this subspace using known methods. The proposed approach works for a wide range of data analysis techniques including k-means clustering, principal component analysis and subspace clustering. The main conceptual contribution is a new coreset definition that allows to charge costs that appear for every solution to an additive constant.
---
paper_title: System of Systems and Big Data analytics - Bridging the gap

paper_content:

Large data has been accumulating in all aspects of our lives for quite some time. Advances in sensor technology, the Internet, wireless communication, and inexpensive memory have all contributed to an explosion of ''Big Data''. System of Systems (SoS) integrate independently operating, non-homogeneous systems to achieve a higher goal than the sum of the parts. Today's SoS are also contributing to the existence of unmanageable ''Big Data''. Recent efforts have developed a promising approach, called ''Data Analytics'', which uses statistical and computational intelligence (CI) tools such as principal component analysis (PCA), clustering, fuzzy logic, neuro-computing, evolutionary computation (such as genetic algorithms), Bayesian networks, etc. to reduce the size of ''Big Data'' to a manageable size and apply these tools to (a) extract information, (b) build a knowledge base using the derived data, and (c) eventually develop a non-parametric model for the ''Big Data''. This paper demonstrates how to construct a bridge between SoS and Data Analytics to develop reliable models for such systems. The subject material for this demonstration is using data analytics to generate a model to forecast produced photovoltaic energy to assist in the optimization of a micro grid SoS. Tools like fuzzy interference, neural networks, PCA, and genetic algorithms are used.
---
paper_title: Principal Component Analysis

paper_content:

Abstract Principal component analysis of a data matrix extracts the dominant patterns in the matrix in terms of a complementary set of score and loading plots. It is the responsibility of the data analyst to formulate the scientific issue at hand in terms of PC projections, PLS regressions, etc. Ask yourself, or the investigator, why the data matrix was collected, and for what purpose the experiments and measurements were made. Specify before the analysis what kinds of patterns you would expect and what you would find exciting. The results of the analysis depend on the scaling of the matrix, which therefore must be specified. Variance scaling, where each variable is scaled to unit variance, can be recommended for general use, provided that almost constant variables are left unscaled. Combining different types of variables warrants blockscaling. In the initial analysis, look for outliers and strong groupings in the plots, indicating that the data matrix perhaps should be “polished” or whether disjoint modeling is the proper course. For plotting purposes, two or three principal components are usually sufficient, but for modeling purposes the number of significant components should be properly determined, e.g. by cross-validation. Use the resulting principal components to guide your continued investigation or chemical experimentation, not as an end in itself.
---
paper_title: Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions

paper_content:

Many problems in computational neuroscience, neuroinformatics, pattern/image recognition, signal processing and machine learning generate massive amounts of multidimensional data with multiple aspects and high dimensionality. Tensors (i.e., multi-way arrays) provide often a natural and compact representation for such massive multidimensional data via suitable low-rank approximations. Big data analytics require novel technologies to efficiently process huge datasets within tolerable elapsed times. Such a new emerging technology for multidimensional big data is a multiway analysis via tensor networks (TNs) and tensor decompositions (TDs) which represent tensors by sets of factor (component) matrices and lower-order (core) tensors. Dynamic tensor analysis allows us to discover meaningful hidden structures of complex data and to perform generalizations by capturing multi-linear and multi-aspect relationships. We will discuss some fundamental TN models, their mathematical and graphical descriptions and associated learning algorithms for large-scale TDs and TNs, with many potential applications including: Anomaly detection, feature extraction, classification, cluster analysis, data fusion and integration, pattern recognition, predictive modeling, regression, time series analysis and multiway component analysis. ::: Keywords: Large-scale HOSVD, Tensor decompositions, CPD, Tucker models, Hierarchical Tucker (HT) decomposition, low-rank tensor approximations (LRA), Tensorization/Quantization, tensor train (TT/QTT) - Matrix Product States (MPS), Matrix Product Operator (MPO), DMRG, Strong Kronecker Product (SKP).
---
paper_title: Learning invariant features through topographic filter maps

paper_content:

Several recently-proposed architectures for high-performance object recognition are composed of two main stages: a feature extraction stage that extracts locally-invariant feature vectors from regularly spaced image patches, and a somewhat generic supervised classifier. The first stage is often composed of three main modules: (1) a bank of filters (often oriented edge detectors); (2) a non-linear transform, such as a point-wise squashing functions, quantization, or normalization; (3) a spatial pooling operation which combines the outputs of similar filters over neighboring regions. We propose a method that automatically learns such feature extractors in an unsupervised fashion by simultaneously learning the filters and the pooling units that combine multiple filter outputs together. The method automatically generates topographic maps of similar filters that extract features of orientations, scales, and positions. These similar filters are pooled together, producing locally-invariant outputs. The learned feature descriptors give comparable results as SIFT on image recognition tasks for which SIFT is well suited, and better results than SIFT on tasks for which SIFT is less well suited.
---
paper_title: Large Scale Distributed Deep Networks

paper_content:

Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider the problem of training a deep network with billions of parameters using tens of thousands of CPU cores. We have developed a software framework called DistBelief that can utilize computing clusters with thousands of machines to train large models. Within this framework, we have developed two algorithms for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas, and (ii) Sandblaster, a framework that supports a variety of distributed batch optimization procedures, including a distributed implementation of L-BFGS. Downpour SGD and Sandblaster L-BFGS both increase the scale and speed of deep network training. We have successfully used our system to train a deep network 30x larger than previously reported in the literature, and achieves state-of-the-art performance on ImageNet, a visual object recognition task with 16 million images and 21k categories. We show that these same techniques dramatically accelerate the training of a more modestly- sized deep network for a commercial speech recognition service. Although we focus on and report performance of these methods as applied to training large neural networks, the underlying algorithms are applicable to any gradient-based machine learning algorithm.
---
paper_title: Gradient-Based Learning Applied to Document Recognition

paper_content:

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.
---
paper_title: An Intelligent Information Forwarder for Healthcare Big Data Systems With Distributed Wearable Sensors

paper_content:

An increasing number of the elderly population wish to live an independent lifestyle, rather than rely on intrusive care programmes. A big data solution is presented using wearable sensors capable of carrying out continuous monitoring of the elderly, alerting the relevant caregivers when necessary and forwarding pertinent information to a big data system for analysis. A challenge for such a solution is the development of context-awareness through the multidimensional, dynamic and nonlinear sensor readings that have a weak correlation with observable human behaviours and health conditions. To address this challenge, a wearable sensor system with an intelligent data forwarder is discussed in this paper. The forwarder adopts a Hidden Markov Model for human behaviour recognition. Locality sensitive hashing is proposed as an efficient mechanism to learn sensor patterns. A prototype solution is implemented to monitor health conditions of dispersed users. It is shown that the intelligent forwarders can provide the remote sensors with context-awareness. They transmit only important information to the big data server for analytics when certain behaviours happen and avoid overwhelming communication and data storage. The system functions unobtrusively, whilst giving the users peace of mind in the knowledge that their safety is being monitored and analysed.
---
paper_title: Big data challenge: a data management perspective

paper_content:

There is a trend that, virtually everyone, ranging from big Web companies to traditional enterprisers to physical science researchers to social scientists, is either already experiencing or anticipating unprecedented growth in the amount of data available in their world, as well as new opportunities and great untapped value. This paper reviews big data challenges from a data management respective. In particular, we discuss big data diversity, big data reduction, big data integration and cleaning, big data indexing and query, and finally big data analysis and mining. Our survey gives a brief overview about big-data-oriented research and problems.
---
paper_title: A Fast Learning Algorithm for Deep Belief Nets

paper_content:

We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
---
paper_title: Reducing the Search Space for Big Data Mining for Interesting Patterns from Uncertain Data

paper_content:

Many existing data mining algorithms search interesting patterns from transactional databases of precise data. However, there are situations in which data are uncertain. Items in each transaction of these probabilistic databases of uncertain data are usually associated with existential probabilities, which express the likelihood of these items to be present in the transaction. When compared with mining from precise data, the search space for mining from uncertain data is much larger due to the presence of the existential probabilities. This problem is worsened as we are moving to the era of Big data. Furthermore, in many real-life applications, users may be interested in a tiny portion of this large search space for Big data mining. Without providing opportunities for users to express the interesting patterns to be mined, many existing data mining algorithms return numerous patterns -- out of which only some are interesting. In this paper, we propose an algorithm that (i) allows users to express their interest in terms of constraints and (ii) uses the MapReduce model to mine uncertain Big data for frequent patterns that satisfy the user-specified constraints. By exploiting properties of the constraints, our algorithm greatly reduces the search space for Big data mining of uncertain data, and returns only those patterns that are interesting to the users for Big data analytics.
---
paper_title: Tradeoffs in scalable data routing for deduplication clusters

paper_content:

As data have been growing rapidly in data centers, deduplication storage systems continuously face challenges in providing the corresponding throughputs and capacities necessary to move backup data within backup and recovery window times. One approach is to build a cluster deduplication storage system with multiple deduplication storage system nodes. The goal is to achieve scalable throughput and capacity using extremely high-throughput (e.g. 1.5 GB/s) nodes, with a minimal loss of compression ratio. The key technical issue is to route data intelligently at an appropriate granularity. ::: ::: We present a cluster-based deduplication system that can deduplicate with high throughput, support deduplication ratios comparable to that of a single system, and maintain a low variation in the storage utilization of individual nodes. In experiments with dozens of nodes, we examine tradeoffs between stateless data routing approaches with low overhead and stateful approaches that have higher overhead but avoid imbalances that can adversely affect deduplication effectiveness for some datasets in large clusters. The stateless approach has been deployed in a two-node commercial system that achieves 3 GB/s for multi-stream deduplication throughput and currently scales to 5.6 PB of storage (assuming 20X total compression).
---
paper_title: Towards Network Reduction on Big Data

paper_content:

The increasing ease of data collection experience and the increasing availability of large data storage space lead to the existence of very large datasets that are commonly referred as "Big Data". Such data not only take over large amount of database storage, but also increase the difficulties for data analysis due to data diversity, which, also makes the datasets seemingly isolated with each other. In this paper, we present a solution to the problem that is to build up connections among the diverse datasets, based upon their similarities. Particularly, a concept of similarity graph along with a similarity graph generation algorithm were introduced. We then proposed a similarity graph reduction algorithm that reduces vertices of the graph for the purpose of graph simplification.
---
paper_title: FlexAnalytics: A flexible data analytics framework for big data applications with I/O performance improvement

paper_content:

Increasingly larger scale applications are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. Instead of moving data from its source to the output storage, in-situ analytics processes output data while simulations are running. However, in-situ data analysis incurs much more computing resource contentions with simulations. Such contentions severely damage the performance of simulation on HPE. Since different data processing strategies have different impact on performance and cost, there is a consequent need for flexibility in the location of data analytics. In this paper, we explore and analyze several potential data-analytics placement strategies along the I/O path. To find out the best strategy to reduce data movement in given situation, we propose a flexible data analytics (FlexAnalytics) framework in this paper. Based on this framework, a FlexAnalytics prototype system is developed for analytics placement. FlexAnalytics system enhances the scalability and flexibility of current I/O stack on HEC platforms and is useful for data pre-processing, runtime data analysis and visualization, as well as for large-scale data transfer. Two use cases – scientific data compression and remote visualization – have been applied in the study to verify the performance of FlexAnalytics. Experimental results demonstrate that FlexAnalytics framework increases data transition bandwidth and improves the application end-to-end transfer performance.
---
paper_title: Characterizing the efficiency of data deduplication for big data storage management

paper_content:

The demand for data storage and processing is increasing at a rapid speed in the big data era. Such a tremendous amount of data pushes the limit on storage capacity and on the storage network. A significant portion of the dataset in big data workloads is redundant. As a result, deduplication technology, which removes replicas, becomes an attractive solution to save disk space and traffic in a big data environment. However, the overhead of extra CPU computation (hash indexing) and IO latency introduced by deduplication should be considered. Therefore, the net effect of using deduplication for big data workloads needs to be examined. To this end, we characterize the redundancy of typical big data workloads to justify the need for deduplication. We analyze and characterize the performance and energy impact brought by deduplication under various big data environments. In our experiments, we identify three sources of redundancy in big data workloads: 1) deploying more nodes, 2) expanding the dataset, and 3) using replication mechanisms. We elaborate on the advantages and disadvantages of different deduplication layers, locations, and granularities. In addition, we uncover the relation between energy overhead and the degree of redundancy. Furthermore, we investigate the deduplication efficiency in an SSD environment for big data workloads.
---
paper_title: Exploiting in-network processing for big data management

paper_content:

Data processing systems face the task of efficiently storing and processing data at petabyte scale, with the amount set to increase in the future. To meet such a requirement, highly scalable, shared-nothing systems, e.g. Google's BigTable [6] or Facebook's Cassandra [14], are built to partition data and process it in parallel on distributed nodes in a cluster. This allows the handling of data at scale but introduces new challenges due to the distribution of data. Running queries involves a high network overhead because data has to be exchanged between cluster nodes and hence, the network becomes a critical part of the system. To avoid the network bottleneck, it is essential for distributed data processing systems (DDPS) to be aware of the network rather than treating it as a black box. We propose in-network processing as a way of achieving network-awareness to decrease bandwidth usage by custom routing, redundancy elimination, and on-path data reduction. Thereby, we can increase the query throughput of a DDPS. The challenges of an in-network processing system range from design issues, such as performance and transparency, to the integration with query optimisation and deployment in data centres. We formulate these challenges as possible research directions and provide a prototype implementation. Our preliminary results suggest that we can significantly improve query throughput in a DDPS by performing partial data reduction within the network.
---
paper_title: Computing persistent features in big data: A distributed dimension reduction approach

paper_content:

Persistent homology has become one of the most popular tools used in topological data analysis for analyzing big data sets. In an effort to minimize the computational complexity of finding the persistent homology of a data set, we develop a simplicial collapse algorithm called the selective collapse. This algorithm works by representing the previously developed strong collapse as a forest and uses that forest data to improve the speed of both the strong collapse and of persistent homology. Finally, we demonstrate the savings in computational complexity using geometric random graphs.
---
paper_title: The Anamorphic Stretch Transform: Putting the Squeeze on “Big Data”

paper_content:

Coping with a deluge of digital information will require more efficient ways to capture, ::: sample and store data. One new approach works by selectively “warping” the data to provide ::: better resolution of the fine details - while still reducing total data size.
---
paper_title: COUPON: A Cooperative Framework for Building Sensing Maps in Mobile Opportunistic Networks

paper_content:

Human-carried or vehicle-mounted sensors can be exploited to collect data ubiquitously for building various sensing maps. Most of existing mobile sensing applications consider users reporting and accessing sensing data through the Internet. However, this approach cannot be applied in the scenarios with poor network coverage or expensive network access. Existing data forwarding schemes for mobile opportunistic networks are not sufficient for sensing applications as spatial-temporal correlation among sensory data has not been explored. In order to build sensing maps satisfying specific sensing quality with low delay and energy consumption, we design COUPON, a novel cooperative sensing and data forwarding framework. We first notice that cooperative sensing scheme can eliminate sampling redundancy and hence save energy. Then we design two cooperative forwarding schemes by leveraging data fusion: Epidemic Routing with Fusion (ERF) and Binary Spray-and-Wait with Fusion (BSWF). Different from previous work assuming that all packets are propagated independently, we consider that packets are spatial-temporal correlated in the forwarding process, and derive the dissemination law of correlated packets. Both the theoretic analysis and simulation results show that our cooperative forwarding schemes can achieve better tradeoff between delivery delay and transmission overhead. We also evaluate our proposed framework and schemes with real mobile traces. Extensive simulations demonstrate that the cooperative sensing scheme can reduce the number of samplings by 93 percent compared with the non-cooperative scheme; ERF can reduce the transmission overhead by 78 percent compared with Epidemic Routing (ER); BSWF can increase the delivery ratio by 16 percent, and reduce the delivery delay and transmission overhead by 5 and 32 percent respectively, compared with Binary Spray-and-Wait (BSW).
---
paper_title: Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis

paper_content:

Higher-order tensors and their decompositions are abundantly present in domains such as signal processing (e.g., higher-order statistics [1] and sensor array processing [2]), scientific computing (e.g., discretized multivariate functions [3]?[6]), and quantum information theory (e.g., representation of quantum many-body states [7]). In many applications, the possibly huge tensors can be approximated well by compact multilinear models or decompositions. Tensor decompositions are more versatile tools than the linear models resulting from traditional matrix approaches. Compared to matrices, tensors have at least one extra dimension. The number of elements in a tensor increases exponentially with the number of dimensions, and so do the computational and memory requirements. The exponential dependency (and the problems that are caused by it) is called the curse of dimensionality. The curse limits the order of the tensors that can be handled. Even for a modest order, tensor problems are often large scale. Large tensors can be handled, and the curse can be alleviated or even removed by using a decomposition that represents the tensor instead of using the tensor itself. However, most decomposition algorithms require full tensors, which renders these algorithms infeasible for many data sets. If a tensor can be represented by a decomposition, this hypothesized structure can be exploited by using compressed sensing (CS) methods working on incomplete tensors, i.e., tensors with only a few known elements.
---
paper_title: Proper orthogonal decomposition based parallel compression for visualizing big data on the K computer

paper_content:

The development of supercomputers has greatly help us to carry on large-scale computing for dealing with various problems through simulating and analyzing them. Visualization is an indispensable tool to understand the properties of the data from supercomputers. Especially, interactive visualization can help us to analyze data from various viewpoints and even to find out some local small but important features. However, it is still difficult to interactively visualize such kind of big data directly due to the slow file I/O problem and the limitation of memory size. For resolving these problems, we proposed a parallel compression method to reduce the data size with low computational cost. Furthermore, the fast linear decompression process is another merit for interactive visualization. Our method uses proper orthogonal decomposition (POD) to compress data because it can effectively extract important features from the data and the resulting compressed data can also be linearly decompressed. Our implementation achieves high parallel efficiency with a binary load-distributed approach, which is similar to the binary-swap image composition used in parallel volume rendering [2]. This approach allows us to effectively utilize all the processing nodes and reduce the interprocessor communication cost throughout the parallel compression calculations. Our test results on the K computer demonstrate superior performance of our design and implementation.
---
paper_title: Analyzing Big Data with Dynamic Quantum Clustering

paper_content:

How does one search for a needle in a multi-dimensional haystack without knowing what a needle is and without knowing if there is one in the haystack? This kind of problem requires a paradigm shift - away from hypothesis driven searches of the data - towards a methodology that lets the data speak for itself. Dynamic Quantum Clustering (DQC) is such a methodology. DQC is a powerful visual method that works with big, high-dimensional data. It exploits variations of the density of the data (in feature space) and unearths subsets of the data that exhibit correlations among all the measured variables. The outcome of a DQC analysis is a movie that shows how and why sets of data-points are eventually classified as members of simple clusters or as members of - what we call - extended structures. This allows DQC to be successfully used in a non-conventional exploratory mode where one searches data for unexpected information without the need to model the data. We show how this works for big, complex, real-world datasets that come from five distinct fields: i.e., x-ray nano-chemistry, condensed matter, biology, seismology and finance. These studies show how DQC excels at uncovering unexpected, small - but meaningful - subsets of the data that contain important information. We also establish an important new result: namely, that big, complex datasets often contain interesting structures that will be missed by many conventional clustering techniques. Experience shows that these structures appear frequently enough that it is crucial to know they can exist, and that when they do, they encode important hidden information. In short, we not only demonstrate that DQC can be flexibly applied to datasets that present significantly different challenges, we also show how a simple analysis can be used to look for the needle in the haystack, determine what it is, and find what this means.
---
paper_title: Large-scale electrophysiology: acquisition, compression, encryption, and storage of big data.

paper_content:

Abstract The use of large-scale electrophysiology to obtain high spatiotemporal resolution brain recordings (>100 channels) capable of probing the range of neural activity from local field potential oscillations to single-neuron action potentials presents new challenges for data acquisition, storage, and analysis. Our group is currently performing continuous, long-term electrophysiological recordings in human subjects undergoing evaluation for epilepsy surgery using hybrid intracranial electrodes composed of up to 320 micro- and clinical macroelectrode arrays. DC-capable amplifiers, sampling at 32 kHz per channel with 18-bits of A/D resolution are capable of resolving extracellular voltages spanning single-neuron action potentials, high frequency oscillations, and high amplitude ultra-slow activity, but this approach generates 3 terabytes of data per day (at 4 bytes per sample) using current data formats. Data compression can provide several practical benefits, but only if data can be compressed and appended to files in real-time in a format that allows random access to data segments of varying size. Here we describe a state-of-the-art, scalable, electrophysiology platform designed for acquisition, compression, encryption, and storage of large-scale data. Data are stored in a file format that incorporates lossless data compression using range-encoded differences, a 32-bit cyclically redundant checksum to ensure data integrity, and 128-bit encryption for protection of patient information.
---
paper_title: A scalable inline cluster deduplication framework for big data protection

paper_content:

Cluster deduplication has become a widely deployed technology in data protection services for Big Data to satisfy the requirements of service level agreement (SLA). However, it remains a great challenge for cluster deduplication to strike a sensible tradeoff between the conflicting goals of scalable deduplication throughput and high duplicate elimination ratio in cluster systems with low-end individual secondary storage nodes. We propose Σ-Dedupe, a scalable inline cluster deduplication framework, as a middleware deployable in cloud data centers, to meet this challenge by exploiting data similarity and locality to optimize cluster deduplication in inter-node and intra-node scenarios, respectively. Governed by a similarity-based stateful data routing scheme, Σ-Dedupe assigns similar data to the same backup server at the super-chunk granularity using a handprinting technique to maintain high cluster-deduplication efficiency without cross-node deduplication, and balances the workload of servers from backup clients. Meanwhile, Σ-Dedupe builds a similarity index over the traditional locality-preserved caching design to alleviate the chunk index-lookup bottleneck in each node. Extensive evaluation of our Σ-Dedupe prototype against state-of-the-art schemes, driven by real-world datasets, demonstrates that Σ-Dedupe achieves a cluster-wide duplicate elimination ratio almost as high as the high-overhead and poorly scalable traditional stateful routing scheme but at an overhead only slightly higher than that of the scalable but low duplicate-elimination-ratio stateless routing approaches.
---
paper_title: Large Scale Distributed Deep Networks

paper_content:

Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider the problem of training a deep network with billions of parameters using tens of thousands of CPU cores. We have developed a software framework called DistBelief that can utilize computing clusters with thousands of machines to train large models. Within this framework, we have developed two algorithms for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas, and (ii) Sandblaster, a framework that supports a variety of distributed batch optimization procedures, including a distributed implementation of L-BFGS. Downpour SGD and Sandblaster L-BFGS both increase the scale and speed of deep network training. We have successfully used our system to train a deep network 30x larger than previously reported in the literature, and achieves state-of-the-art performance on ImageNet, a visual object recognition task with 16 million images and 21k categories. We show that these same techniques dramatically accelerate the training of a more modestly- sized deep network for a commercial speech recognition service. Although we focus on and report performance of these methods as applied to training large neural networks, the underlying algorithms are applicable to any gradient-based machine learning algorithm.
---
paper_title: Computing Topological Persistence for Simplicial Maps

paper_content:

Algorithms for persistent homology are well-studied where homomorphisms are induced by inclusion maps. In this paper, we propose a practical algorithm for computing persistence under Z2 coefficients for a (monotone) sequence of general simplicial maps and show how these maps arise naturally in some applications of topological data analysis. ::: A simplicial map can be decomposed into a set of elementary inclusions and vertex collapses--two atomic operations that can be supported efficiently with the notion of simplex annotations for computing persistent homology. A consistent annotation through these atomic operations implies the maintenance of a consistent cohomology basis, hence a homology basis by duality. While the idea of maintaining a cohomology basis through an inclusion is not new, maintaining them through a vertex collapse is new, which constitutes an important atomic operation for simulating simplicial maps. Annotations support the vertex collapse in addition to the usual inclusion quite naturally. ::: Finally, we exhibit an application of this new tool in which we approximate the persistence diagram of a filtration of Rips complexes where vertex collapses are used to tame the blow-up in size.
---
paper_title: Reduced topologically real-world networks: a big-data approach

paper_content:

The topological and dynamical properties of real-world networks have attracted extensive research from a variety of multi-disciplinary fields. They, in fact, model typically big datasets which pose interesting challenges, due to their intrinsic size and complex interactions, as well as the dependencies between their different sub-parts. Therefore, defining networks based on such properties, is unlikely to produce usable information due to their complexity and the data inconsistencies which they typically contain. In this paper, the authors discuss the evaluation of a method as part of ongoing research which aims to mine data to assess whether their associated networks exhibit properties comparable to well-known structures, namely scale-free, small world and random networks. For this, they will use a large dataset containing information on the seismologic activity recorded by the European-Mediterranean Seismological Centre. The authors will show that it provides an accurate, agile, and scalable tool to extract useful information. This further motivates their effort to produce a big data analytics tool which will focus on obtaining in-depth intelligence from both structured and unstructured big datasets. This will ultimately lead to a better understanding and prediction of the properties of the system(s) they model.
---
paper_title: An influence assessment method based on co-occurrence for topologically reduced big data sets

paper_content:

The extraction of meaningful, accurate, and relevant information is at the core of Big Data research. Furthermore, the ability to obtain an insight is essential in any decision-making process, even though the diverse and complex nature of big data sets raises a multitude of challenges. In this paper, we propose a novel method to address the automated assessment of influence among concepts in big data sets. This is carried out by investigating their mutual co-occurrence, which is determined via topologically reducing the corresponding network. The main motivation is to provide a toolbox to classify and analyse influence properties, which can be used to investigate their dynamical and statistical behaviour, potentially leading to a better understanding and prediction of the properties of the system(s) they model. An evaluation was carried out on two real-world data sets, which were analysed to test the capabilities of our system. The results show the potential of our approach, indicating both accuracy and efficiency.
---
paper_title: Statistical wavelet-based anomaly detection in big data with compressive sensing

paper_content:

Anomaly detection in big data is a key problem in the big data analytics domain. In this paper, the definitions of anomaly detection and big data were presented. Due to the sampling and storage burden and the inadequacy of privacy protection of anomaly detection based on uncompressed data, compressive sensing theory was introduced and used in the anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The proposed anomaly detection technique was used for through-wall human detection to demonstrate the effectiveness. The experiments for detecting humans behind a brick wall and gypsum based on ultra-wideband radar signal were carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of a human being through compressed signals and uncompressed data.
---
paper_title: An Intelligent Information Forwarder for Healthcare Big Data Systems With Distributed Wearable Sensors

paper_content:

An increasing number of the elderly population wish to live an independent lifestyle, rather than rely on intrusive care programmes. A big data solution is presented using wearable sensors capable of carrying out continuous monitoring of the elderly, alerting the relevant caregivers when necessary and forwarding pertinent information to a big data system for analysis. A challenge for such a solution is the development of context-awareness through the multidimensional, dynamic and nonlinear sensor readings that have a weak correlation with observable human behaviours and health conditions. To address this challenge, a wearable sensor system with an intelligent data forwarder is discussed in this paper. The forwarder adopts a Hidden Markov Model for human behaviour recognition. Locality sensitive hashing is proposed as an efficient mechanism to learn sensor patterns. A prototype solution is implemented to monitor health conditions of dispersed users. It is shown that the intelligent forwarders can provide the remote sensors with context-awareness. They transmit only important information to the big data server for analytics when certain behaviours happen and avoid overwhelming communication and data storage. The system functions unobtrusively, whilst giving the users peace of mind in the knowledge that their safety is being monitored and analysed.
---
paper_title: An Analytical Tool to Map Big Data to Networks with Reduced Topologies

paper_content:

The topological and dynamical properties of real-world networks have attracted extensive research from a variety of multi-disciplinary fields. They, in fact, model typically big datasets which pose interesting challenges, due to their intrinsic size and complex interactions, as well as the dependencies between their different sub-parts. Therefore, defining networks based on such properties, is unlikely to produce usable information due to their complexity and the data inconsistencies which they typically contain. In this paper, we discuss the evaluation of a method as part of ongoing research which aims to mine data to assess whether their associated networks exhibit properties comparable to well-known structures, namely scale-free, small world and random networks. For this, we will use a large dataset containing information on the seismologic activity recorded by the European-Mediterranean Seismological Centre. We will show that it provides an accurate, agile, and scalable tool to extract useful information. This further motivates our effort to produce a big data analytics tool which will focus on obtaining in-depth intelligence from both structured and unstructured big datasets. This will ultimately lead to a better understanding and prediction of the properties of the system(s) they model.
---
paper_title: A spatiotemporal compression based approach for efficient big data processing on Cloud

paper_content:

It is well known that processing big graph data can be costly on Cloud. Processing big graph data introduces complex and multiple iterations that raise challenges such as parallel memory bottlenecks, deadlocks, and inefficiency. To tackle the challenges, we propose a novel technique for effectively processing big graph data on Cloud. Specifically, the big data will be compressed with its spatiotemporal features on Cloud. By exploring spatial data correlation, we partition a graph data set into clusters. In a cluster, the workload can be shared by the inference based on time series similarity. By exploiting temporal correlation, in each time series or a single graph edge, temporal data compression is conducted. A novel data driven scheduling is also developed for data processing optimisation. The experiment results demonstrate that the spatiotemporal compression and scheduling achieve significant performance gains in terms of data size and data fidelity loss.
---
paper_title: Deep learning via Hessian-free optimization

paper_content:

We develop a 2nd-order optimization method based on the "Hessian-free" approach, and apply it to training deep auto-encoders. Without using pre-training, we obtain results superior to those reported by Hinton & Salakhutdinov (2006) on the same tasks they considered. Our method is practical, easy to use, scales nicely to very large datasets, and isn't limited in applicability to auto-encoders, or any specific model class. We also discuss the issue of "pathological curvature" as a possible explanation for the difficulty of deep-learning and how 2nd-order optimization, and our method in particular, effectively deals with it.
---
paper_title: Reducing the Search Space for Big Data Mining for Interesting Patterns from Uncertain Data

paper_content:

Many existing data mining algorithms search interesting patterns from transactional databases of precise data. However, there are situations in which data are uncertain. Items in each transaction of these probabilistic databases of uncertain data are usually associated with existential probabilities, which express the likelihood of these items to be present in the transaction. When compared with mining from precise data, the search space for mining from uncertain data is much larger due to the presence of the existential probabilities. This problem is worsened as we are moving to the era of Big data. Furthermore, in many real-life applications, users may be interested in a tiny portion of this large search space for Big data mining. Without providing opportunities for users to express the interesting patterns to be mined, many existing data mining algorithms return numerous patterns -- out of which only some are interesting. In this paper, we propose an algorithm that (i) allows users to express their interest in terms of constraints and (ii) uses the MapReduce model to mine uncertain Big data for frequent patterns that satisfy the user-specified constraints. By exploiting properties of the constraints, our algorithm greatly reduces the search space for Big data mining of uncertain data, and returns only those patterns that are interesting to the users for Big data analytics.
---
paper_title: Malicious URL filtering — A big data application

paper_content:

Malicious URLs have become a channel for Internet criminal activities such as drive-by-download, spamming and phishing. Applications for the detection of malicious URLs are accurate but slow (because they need to download the content or query some Internet host information). In this paper we present a novel lightweight filter based only on the URL string itself to use before existing processing methods. We run experiments on a large dataset and demonstrate a 75% reduction in workload size while retaining at least 90% of malicious URLs. Existing methods do not scale well with the hundreds of millions of URLs encountered every day as the problem is a heavily-imbalanced, large-scale binary classification problem. Our proposed method is able to handle nearly two million URLs in less than five minutes. We generate two filtering models by using lexical features and descriptive features, and then combine the filtering results. The on-line learning algorithms are applied here not only for dealing with large-scale data sets but also for fitting the very short lifetime characteristics of malicious URLs. Our filter can significantly reduce the volume of URL queries on which further analysis needs to be performed, saving both computing time and bandwidth used for content retrieval.
---
paper_title: Privacy-Preserving Distributed Movement Data Aggregation

paper_content:

We propose a novel approach to privacy-preserving analytical processing within a distributed setting, and tackle the problem of obtaining aggregated information about vehicle traffic in a city from movement data collected by individual vehicles and shipped to a central server. Movement data are sensitive because people’s whereabouts have the potential to reveal intimate personal traits, such as religious or sexual preferences, and may allow re-identification of individuals in a database. We provide a privacy-preserving framework for movement data aggregation based on trajectory generalization in a distributed environment. The proposed solution, based on the differential privacy model and on sketching techniques for efficient data compression, provides a formal data protection safeguard. Using real-life data, we demonstrate the effectiveness of our approach also in terms of data utility preserved by the data transformation.
---

Title: Big Data Reduction Methods: A Survey

Section 1: Introduction
Description 1: This section introduces the concept of big data and the necessity for data reduction methods due to various characteristics such as volume, velocity, and dimensionality.

Section 2: Big Data Complexity and the Need for Data Reduction
Description 2: This section discusses the inherent complexities in big data systems and the critical need for data reduction to manage and analyze large-scale data effectively.

Section 3: Big Data Reduction Methods
Description 3: This section presents various methods for reducing big data, including network theory, compression, data deduplication, and redundancy elimination.

Section 4: Network Theory
Description 4: This subsection details how network (or graph) theory is used to reduce high-dimensional unstructured data into more manageable, low-dimensional structured data.

Section 5: Compression
Description 5: This subsection explores different compression-based methods to reduce the size of datasets while preserving the original data streams.

Section 6: Data Deduplication (Redundancy Elimination)
Description 6: This subsection covers techniques for removing redundant data to improve storage efficiency and processing speed in big data environments.

Section 7: Data Preprocessing
Description 7: This subsection discusses methods for preprocessing data before storage to reduce data size and enhance data quality.

Section 8: Dimension Reduction
Description 8: This subsection focuses on techniques to reduce the number of features or dimensions in datasets to simplify analysis and storage.

Section 9: Data Mining and Machine Learning (DM and ML)
Description 9: This subsection highlights the use of data mining and machine learning methods for reducing and extracting meaningful patterns from big data.

Section 10: Open Research Issues
Description 10: This section outlines the current open research issues, limitations of existing methods, and potential future research directions in the field of big data reduction.

Section 11: Conclusions
Description 11: This section summarizes the key findings of the survey and emphasizes the importance of continued research in big data reduction methods.