Algorithm and approaches to handle large Data- A Survey
---
paper_title: Design Principles for Effective Knowledge Discovery from Big Data

paper_content:

Big data phenomenon refers to the practice of collection and processing of very large data sets and associated systems and algorithms used to analyze these massive datasets. Architectures for big data usually range across multiple machines and clusters, and they commonly consist of multiple special purpose sub-systems. Coupled with the knowledge discovery process, big data movement offers many unique opportunities for organizations to benefit (with respect to new insights, business optimizations, etc.). However, due to the difficulty of analyzing such large datasets, big data presents unique systems engineering and architectural challenges. In this paper, we present three system design principles that can inform organizations on effective analytic and data collection processes, system organization, and data dissemination practices. The principles presented derive from our own research and development experiences with big data problems from various federal agencies, and we illustrate each principle with our own experiences and recommendations.
---
paper_title: Considerations for big data: Architecture and approach

paper_content:

The amount of data in our industry and the world is exploding. Data is being collected and stored at unprecedented rates. The challenge is not only to store and manage the vast volume of data (“big data”), but also to analyze and extract meaningful value from it. There are several approaches to collecting, storing, processing, and analyzing big data. The main focus of the paper is on unstructured data analysis. Unstructured data refers to information that either does not have a pre-defined data model or does not fit well into relational tables. Unstructured data is the fastest growing type of data, some example could be imagery, sensors, telemetry, video, documents, log files, and email data files. There are several techniques to address this problem space of unstructured analytics. The techniques share a common character tics of scale-out, elasticity and high availability. MapReduce, in conjunction with the Hadoop Distributed File System (HDFS) and HBase database, as part of the Apache Hadoop project is a modern approach to analyze unstructured data. Hadoop clusters are an effective means of processing massive volumes of data, and can be improved with the right architectural approach.
---
paper_title: An Efficient Technique on Cluster Based Master Slave Architecture Design

paper_content:

We are in an age repeatedly referred to the information age. In this age, because we suppose that information leads to power and achievement, and credit to sophisticated techniques such as computers, satellites etc. we include collecting large amount of information similar to business transaction, scientific data, medical data satellite data, surveillance video a pictures world wide web and many more. With the enormous amount of data stored in files, data base and this technique hold large amount of data is called data mining. Data mining is the method of extracting valuable information from the enormous amount of data saved in the database and files. At that time basically two important reasons that are used in data mining. First our capacity to accumulate and store the large amount of data is fast increase day by day, and second but the most imperative basis is the need to turn such data into useful information and knowledge. Association rule mining is a vital procedure to find out hidden interaction with items in the transaction. This paper describes new technique cluster based Master-Slave architecture. It uses Hybrid technique, the grouping of bottom up and top down technique for search repetitive item sets. It reduces the time in use to get out the support count of the item sets. The Prime number show present offers the give for verify rules and provides decrease in the data complexity.
---
paper_title: Compression, clustering, and pattern discovery in very high-dimensional discrete-attribute data sets

paper_content:

This paper presents an efficient framework for error-bounded compression of high-dimensional discrete-attribute data sets. Such data sets, which frequently arise in a wide variety of applications, pose some of the most significant challenges in data analysis. Subsampling and compression are two key technologies for analyzing these data sets. The proposed framework, PROXIMUS, provides a technique for reducing large data sets into a much smaller set of representative patterns, on which traditional (expensive) analysis algorithms can be applied with minimal loss of accuracy. We show desirable properties of PROXIMUS in terms of runtime, scalability to large data sets, and performance in terms of capability to represent data in a compact form and discovery and interpretation of interesting patterns. We also demonstrate sample applications of PROXIMUS in association rule mining and semantic classification of term-document matrices. Our experimental results on real data sets show that use of the compressed data for association rule mining provides excellent precision and recall values (above 90 percent) across a range of problem parameters while reducing the time required for analysis drastically. We also show excellent interpretability of the patterns discovered by PROXIMUS in the context of clustering and classification of terms and documents. In doing so, we establish PROXIMUS as a tool for both preprocessing data before applying computationally expensive algorithms and directly extracting correlated patterns.
---
paper_title: Hierarchical Artificial Neural Networks for Recognizing High Similar Large Data Sets

paper_content:

This paper proposes a hierarchical artificial neural network for recognizing high similar large data sets. It is usually required to classify large data sets with high similar characteristics in many applications. Analyzing and identifying those data is a laborious task when the methods adopted are primarily based on visual inspection. In many field applications, data sets are measured and recorded continuously using automatic monitoring equipments. Therefore, a large amount of data can be collected, and manual inspection has become an unsuitable approach to recognizing those data. This proposed hierarchical neural network integrates self-organizing feature map (SOM) networks and learning vector quantization (LVQ) networks. The SOM networks provide an approximate method for computing the input vectors in an unsupervised manner. Then the computation of the SOM may be viewed as the first stage of the proposed hierarchical network. The second stage is provided by the LVQ networks based on a supervised learning technique that uses class information to improve the quality of the classifier from the first stage. The multistage hierarchical network attempts to factorize the overall input vector into a number of small groups, each of which requires very little computation. Consequently, by use of the proposed network, the loss in accuracy can be small.
---
paper_title: Online Mining of data to generate association rule mining in large databases

paper_content:

Data Mining is a Technology to explore data, analyze the data and finally discovering patterns from large data repository. In this paper, the problem of online mining of association rules in large databases is discussed. Online association rule mining can be applied which helps to remove redundant rules and helps in compact representation of rules for user. In this paper, a new and more optimized algorithm has been proposed for online rule generation. The advantage of this algorithm is that the graph generated in our algorithm has less edge as compared to the lattice used in the existing algorithm. The Proposed algorithm generates all the essential rules also and no rule is missing. The use of non redundant association rules help significantly in the reduction of irrelevant noise in the data mining process. This graph theoretic approach, called adjacency lattice is crucial for online mining of data. The adjacency lattice could be stored either in main memory or secondary memory. The idea of adjacency lattice is to pre store a number of large item sets in special format which reduces disc I/O required in performing the query.
---
paper_title: Streaming Hierarchical Clustering for Concept Mining

paper_content:

We are concerned with the general problem of concept mining - discovering useful associations, relationships, and groupings in large collections of data. Mathematical transformation algorithms have proven effective at reducing the content of multilingual, unstructured data into a vector that describes the content. Such methods are particularly desirable in fields undergoing information explosions, such as network traffic analysis, bioinformatics, and the intelligence community. In response, concept mining methodology is being extended to improve performance and permit hardware implementation -traditional methods are not sufficiently scalable. Hardware-accelerated systems have proven effective at automatically classifying such content when topics are known in advance. Our complete system builds on our past work in this area, presented in the Aerospace 2005 and 2006 conferences, where we described a novel algorithmic approach for extracting semantic content from unstructured text document streams. However, there is an additional need within the intelligence community to cluster related sets of content without advance training. To allow this function to happen at high speed, we have implemented a system that hierarchically clusters streaming content. The method, streaming hierarchical partitioning, is designed to be implemented in hardware and handle extremely high ingestion rates. As new documents are ingested, they are dynamically organized into a hierarchy, which has a fixed maximal size. Once this limit is reached, documents must consequently be excreted at a rate equaling their ingestion. The choice of documents to excrete is a point of interest -we present several autonomous heuristics for doing so intelligently, as well as a proposal for incorporating user interaction to focus attention on concepts of interest. A related desideratum is robust accommodation of concept drift -gradual change in the distribution and content of the document stream over time. Accordingly, we present and analyze experimental results for document streams evolving over time under several regimes. Current and proposed methods for concisely and informatively presenting derived content from streaming hierarchical clustering to the user for analysis are presented in this content. To support our claims of eventual hardware implementation and real-time performance with a high ingestion rate, we provide a detailed hardware-ready design, with asymptotic analysis and performance predictions. The system has been prototyped and tested on a Xeon processor as well as on a PowerPC embedded within a Xilinx Virtex2 FPGA. In summary, we describe a system designed to satisfy three primary goals: (1) real-time concept mining of high-volume data streams; (2) dynamic organization of concepts into a relational hierarchy; (3) adaptive reorganization of the concept hierarchy in response to evolving circumstances and user feedback.
---
paper_title: Data Field for Hierarchical Clustering

paper_content:

In this paper, data field is proposed to group data objects via simulating their mutual interactions and opposite movements for hierarchical clustering. Enlightened by the field in physical space, data field to simulate nuclear field is presented to illuminate the interaction between objects in data space. In the data field, the self-organized process of equipotential lines on many data objects discovers their hierarchical clustering-characteristics. During the clustering process, a random sample is first generated to optimize the impact factor. The masses of data objects are then estimated to select core data object with nonzero masses. Taking the core data objects as the initial clusters, the clusters are iteratively merged hierarchy by hierarchy with good performance. The results of a case study show that the data field is capable of hierarchical clustering on objects varying size, shape or granularity without user-specified parameters, as well as considering the object features inside the clusters and removing the outliers from noisy data. The comparisons illustrate that the data field clustering performs better than K-means, BIRCH, CURE, and CHAMELEON.
---
paper_title: Analyzing large biological datasets with association networks

paper_content:

Due to advances in high-throughput biotechnologies biological information is being collected in databases at an amazing rate, requiring novel computational approaches that process collected data into new knowledge in a timely manner. In this study, we propose a computational framework for discovering modular structure, relationships and regularities in complex data. The framework utilizes a semantic-preserving vocabulary to convert records of biological annotations of an object, such as an organism, gene, chemical or sequence, into networks (Anets) of the associated annotations. An association between a pair of annotations in an Anet is determined by the similarity of their co-occurrence pattern with all other annotations in the data. This feature captures associations between annotations that do not necessarily co-occur with each other and facilitates discovery of the most significant relationships in the collected data through clustering and visualization of the Anet. To demonstrate this approach, we applied the framework to the analysis of metadata from the Genomes OnLine Database and produced a biological map of sequenced prokaryotic organisms with three major clusters of metadata that represent pathogens, environmental isolates and plant symbionts.
---
paper_title: Decision tree learning on very large data sets

paper_content:

Consider a labeled data set of 1 terabyte in size. A salient subset might depend upon the users interests. Clearly, browsing such a large data set to find interesting areas would be very time consuming. An intelligent agent which, for a given class of user, could provide hints on areas of the data that might interest the user would be very useful. Given large data sets having categories of salience for different user classes attached to the data in them, these labeled sets of data can be used to train a decision tree to label unseen data examples with a category of salience. The training set will be much larger than usual. This paper describes an approach to generating the rules for an agent from a large training set. A set of decision trees are built in parallel on tractable size training data sets which are a subset of the original data. Each learned decision tree will be reduced to a set of rules, conflicting rules resolved and the resultant rules merged into one set. Results from cross validation experiments on a data set suggest this approach may be effectively applied to large sets of data.
---
paper_title: A Hybrid Evolutionary Approach To Construct Optimal Decision Trees With Large Data Sets

paper_content:

Data mining environments produces large volume of data. The large amount of knowledge contains can be utilized to improve decision-making process of an organization. Large amount of available data when used for decision tree construction builds large sized trees that are incomprehensible to human experts. The learning process on this high volume data becomes very slow, as it has to be done serially on available large datasets. Our ultimate goal is to build smaller trees with equally accurate solutions with randomly selected sampled data. We experimented on techniques based on the idea of incremental random sampling combined with genetic algorithms that uses global search techniques to evolve decision Trees to obtain compact representation of large data set. Experiments performed on some data sets proved that the proposed random sampling procedures with genetic algorithms to build decision Trees gives relatively smaller trees as compared to other methods but equally accurate solution as other methods. The method incorporates optimization with the comprehensibility and scalability. We tried to explore the method using that we can avoid problems like slow execution, overloading of memory and processor with very large database can be avoided using the technique.
---
paper_title: Detecting novel associations in large data sets

paper_content:

Identifying interesting relationships between pairs of variables in large data sets is increasingly important. Here, we present a measure of dependence for two-variable relationships: the maximal information coefficient (MIC). MIC captures a wide range of associations both functional and not, and for functional relationships provides a score that roughly equals the coefficient of determination ( R 2 ) of the data relative to the regression function. MIC belongs to a larger class of maximal information-based nonparametric exploration (MINE) statistics for identifying and classifying relationships. We apply MIC and MINE to data sets in global health, gene expression, major-league baseball, and the human gut microbiota and identify known and novel relationships.
---
paper_title: A clustering method for very large mixed data sets

paper_content:

In developed countries, especially over the last decade, there has been an explosive growth in the capability to generate, collect and use very large data sets. The objects of these data sets could be simultaneously described by quantitative and qualitative attributes. At present, algorithms able to process either very large data sets (in metric spaces) or mixed (qualitative and quantitative) incomplete data (missing value) sets have been developed, but not for very large mixed incomplete data sets. In this paper we introduce a new clustering method named GLC+ to process very large mixed incomplete data sets in order to obtain a partition in connected sets.
---
paper_title: Hashing scheme for space-efficient detection and localization of changes in large data sets

paper_content:

One of primary components of computer systems security is preservation of data integrity. In addition to violation prevention, it also includes methods used to detect such violations after they occur. The most common methods for preserving integrity of binary data are based on various hashing functions. However, an inherent downside of using hashing functions is that a single hash can only be used to verify integrity of a single data string as a whole. This results in it being impossible to locate the exact position within the string where the change occurred. Alternative method entails splitting the data string into blocks, each protected by a hash. While this enables more precise location of changes, storing potentially large number of hashes imposes significant space overhead for large data sets. In this paper, we present a space-efficient method for detection and localization of unwanted changes in large data sets. Our method reuses the idea of splitting data into blocks and hashing each block separately, but with certain added properties: logarithmic instead of linear increase of memory space required to store hashes, ease of parallelization, possible application on distributed data, limited self-verification of hashes and efficient recalculation of hashes for dynamically changing data.
---
paper_title: Big data privacy issues in public social media

paper_content:

Big Data is a new label given to a diverse field of data intensive informatics in which the datasets are so large that they become hard to work with effectively. The term has been mainly used in two contexts, firstly as a technological challenge when dealing with dataintensive domains such as high energy physics, astronomy or internet search, and secondly as a sociological problem when data about us is collected and mined by companies such as Facebook, Google, mobile phone companies, retail chains and governments. In this paper we look at this second issue from a new perspective, namely how can the user gain awareness of the personally relevant part Big Data that is publicly available in the social web. The amount of user-generated media uploaded to the web is expanding rapidly and it is beyond the capabilities of any human to sift through it all to see which media impacts our privacy. Based on an analysis of social media in Flickr, Locr, Facebook and Google+, we discuss privacy implications and potential of the emerging trend of geo-tagged social media. We then present a concept with which users can stay informed about which parts of the social Big Data deluge is relevant to them.
---
paper_title: A Metadata Based Storage Model for Securing Data in Cloud Environment

paper_content:

IT Enterprises are migrating to the Cloud environment at a faster pace. Though Cloud Computing is quickly evolving as the next generation architecture for enterprises, there are astounding issues with this environment that will pop up as more and more applications and data move into the cloud. Security of information that is being processed by the applications and ultimately getting stored in the data centers are of big concerns of this newly evolving environment. The security of the data is a concern not only during transferring of data through the wires but also during its storage. And the architecture that is needed to secure the stored data is of much importance than while the data is getting transferred because of the fact that the data resides relatively for a long time in the storage area than in the wires. To ensure the security of the data stored in the data centers, we propose a new methodology which might not completely help in restricting a hacker to access the data but will make the data invaluable if it is extracted by a hacker but at the same time ensures the quality of the data that is being provided to its respective owner or authorized user. We propose a metadata based data segregation and storage methodology and also solutions to access this segregated data. This methodology ensures that data is invaluable during static residence and gains value only during acquisition or updation.
---

Title: Algorithm and Approaches to Handle Large Data - A Survey

Section 1: Introduction
Description 1: Introduce the concept of data mining, big data, and the significance of handling large datasets.

Section 2: Architecture
Description 2: Describe the architecture of big data systems, including storage, handling, and analysis.

Section 3: Algorithm
Description 3: Discuss various algorithms used to process and analyze big data.

Section 4: Issues with Big Data
Description 4: Discuss the security and privacy issues associated with big data, as well as solutions to address these concerns.

Section 5: Conclusion
Description 5: Summarize the methodologies, architectures, algorithms, and issues in handling large datasets, and the importance of formalizing big data analysis processes.