A Review of Polyglot Persistence in the Big Data World
---
paper_title: "One size fits all": an idea whose time has come and gone

paper_content:

The last 25 years of commercial DBMS development can be summed up in a single phrase: "one size fits all". This phrase refers to the fact that the traditional DBMS architecture (originally designed and optimized for business data processing) has been used to support many data-centric applications with widely varying characteristics and requirements. In this paper, we argue that this concept is no longer applicable to the database market, and that the commercial world will fracture into a collection of independent database engines, some of which may be unified by a common front-end parser. We use examples from the stream-processing market and the data-warehouse market to bolster our claims. We also briefly discuss other markets for which the traditional architecture is a poor fit and argue for a critical rethinking of the current factoring of systems services into products.
---
paper_title: Big data: The next frontier for innovation, competition, and productivity

paper_content:

The amount of data in our world has been exploding, and analyzing large data sets—so-called big data— will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.
---
paper_title: Warehousing and Protecting Big Data: State-Of-The-Art-Analysis, Methodologies, Future Challenges

paper_content:

This paper proposes a comprehensive critical survey on the issues of warehousing and protecting big data, which are recognized as critical challenges of emerging big data research. Indeed, both are critical aspects to be considered in order to build truly, high-performance and highly-flexible big data management systems. We report on state-of-the-art approaches, methodologies and trends, and finally conclude by providing open problems and challenging research directions to be considered by future efforts.
---
paper_title: Big Data: Principles and Paradigms

paper_content:

Big Data: Principles and Paradigms captures the state-of-the-art research on the architectural aspects, technologies, and applications of Big Data. The book identifies potential future directions and technologies that facilitate insight into numerous scientific, business, and consumer applications. To help realize Big Datas full potential, the book addresses numerous challenges, offering the conceptual and technological solutions for tackling them. These challenges include life-cycle data management, large-scale storage, flexible processing infrastructure, data modeling, scalable machine learning, data analysis algorithms, sampling techniques, and privacy and ethical issues. Covers computational platforms supporting Big Data applications Addresses key principles underlying Big Data computing Examines key developments supporting next generation Big Data platforms Explores the challenges in Big Data computing and ways to overcome them Contains expert contributors from both academia and industry
---
paper_title: A relational model of data for large shared data banks

paper_content:

Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). A prompting service which supplies such information is not a satisfactory solution. Activities of users at terminals and most application programs should remain unaffected when the internal representation of data is changed and even when some aspects of the external representation are changed. Changes in data representation will often be needed as a result of changes in query, update, and report traffic and natural growth in the types of stored information. Existing noninferential, formatted data systems provide users with tree-structured files or slightly more general network models of the data. In Section 1, inadequacies of these models are discussed. A model based on n-ary relations, a normal form for data base relations, and the concept of a universal data sublanguage are introduced. In Section 2, certain operations on relations (other than logical inference) are discussed and applied to the problems of redundancy and consistency in the user's model.
---
paper_title: Big-data: Acid versus base for database transactions

paper_content:

Database developers all know the ACID acronym. It says that database transactions should be: Atomic, Consistent, Isolated, and Durable. These qualities seem indispensable, and yet they are incompatible with availability and performance in very large systems. For example, suppose you run an online book store and you proudly display how many of each book you have in your inventory. Every time someone is in the process of buying a book, you lock part of the database until they finish so that all visitors around the world will see accurate inventory numbers. That works well if you run The Shop around the Corner but not if you run Amazon.com. Amazon might instead use cached data. Users would not see not the inventory count at this second, but what it was say an hour ago when the last snapshot was taken. Also, Amazon might violate the “I” in ACID by tolerating a small probability that simultaneous transactions could interfere with each other. For example, two customers might both believe that they just purchased the last copy of a certain book. The company might risk having to apologize to one of the two customers (and maybe compensate them with a gift card) rather than slowing down their site and irritating myriad other customers. There is a computer science theorem that quantifies the inevitable trade-offs. Eric Brewer's CAP theorem says that if you want consistency, availability, and partition tolerance, you have to settle for two out of three. (For a distributed system, partition tolerance means the system will continue to work unless there is a total network failure. A few nodes can fail and the system keeps going.) An alternative to ACID is BASE: Basic Availability, Soft-state, Eventual consistency.
---
paper_title: Principles of transaction-oriented database recovery

paper_content:

In this paper, a terminological framework is provided for describing different transactionoriented recovery schemes for database systems in a conceptual rather than an implementation-dependent way. By introducing the terms materialized database, propagation strategy, and checkpoint, we obtain a means for classifying arbitrary implementations from a unified viewpoint. This is complemented by a classification scheme for logging techniques, which are precisely defined by using the other terms. It is shown that these criteria are related to all relevant questions such as speed and scope of recovery and amount of redundant information required. The primary purpose of this paper, however, is to establish an adequate and precise terminology for a topic in which the confusion of concepts and implementational aspects still imposes a lot of problems.
---
paper_title: Hive - A Warehousing Solution Over a Map-Reduce Framework

paper_content:

The size of data sets being collected and analyzed in the industry for business intelligence is growing rapidly, making traditional warehousing solutions prohibitively expensive. Hadoop [3] is a popular open-source map-reduce implementation which is being used as an alternative to store and process extremely large data sets on commodity hardware. However, the map-reduce programming model is very low level and requires developers to write custom programs which are hard to maintain and reuse.
---
paper_title: Hive - a petabyte scale data warehouse using Hadoop

paper_content:

The size of data sets being collected and analyzed in the industry for business intelligence is growing rapidly, making traditional warehousing solutions prohibitively expensive. Hadoop [1] is a popular open-source map-reduce implementation which is being used in companies like Yahoo, Facebook etc. to store and process extremely large data sets on commodity hardware. However, the map-reduce programming model is very low level and requires developers to write custom programs which are hard to maintain and reuse. In this paper, we present Hive, an open-source data warehousing solution built on top of Hadoop. Hive supports queries expressed in a SQL-like declarative language - HiveQL, which are compiled into map-reduce jobs that are executed using Hadoop. In addition, HiveQL enables users to plug in custom map-reduce scripts into queries. The language includes a type system with support for tables containing primitive types, collections like arrays and maps, and nested compositions of the same. The underlying IO libraries can be extended to query data in custom formats. Hive also includes a system catalog - Metastore - that contains schemas and statistics, which are useful in data exploration, query optimization and query compilation. In Facebook, the Hive warehouse contains tens of thousands of tables and stores over 700TB of data and is being used extensively for both reporting and ad-hoc analyses by more than 200 users per month.
---
paper_title: The Google file system

paper_content:

We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.
---
paper_title: NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence

paper_content:

The need to handle increasingly larger data volumes is one factor driving the adoption of a new class of nonrelational NoSQL databases. Advocates of NoSQL databases claim they can be used to build systems that are more performant, scale better, and are easier to program. NoSQL Distilled is a concise but thorough introduction to this rapidly emerging technology. Pramod J. Sadalage and Martin Fowler explain how NoSQL databases work and the ways that they may be a superior alternative to a traditional RDBMS. The authors provide a fast-paced guide to the concepts you need to know in order to evaluate whether NoSQL databases are right for your needs and, if so, which technologies you should explore further. The first part of the book concentrates on core concepts, including schemaless data models, aggregates, new distribution models, the CAP theorem, and map-reduce. In the second part, the authors explore architectural and design issues associated with implementing NoSQL. They also present realistic use cases that demonstrate NoSQL databases at work and feature representative examples using Riak, MongoDB, Cassandra, and Neo4j. In addition, by drawing on Pramod Sadalages pioneering work, NoSQL Distilled shows how to implement evolutionary design with schema migration: an essential technique for applying NoSQL databases. The book concludes by describing how NoSQL is ushering in a new age of Polyglot Persistence, where multiple data-storage worlds coexist, and architects can choose the technology best optimized for each type of data access.
---
paper_title: The Hadoop Distributed File System

paper_content:

The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.
---
paper_title: A comparison between several NoSQL databases with comments and notes

paper_content:

This paper is trying to comment on the various NoSQL (Not only Structured Query Language) systems and to make a comparison (using multiple criteria) between them. The NoSQL databases were created as a mean to offer high performance (both in terms of speed and size) and high availability at the price of loosing the ACID (Atomic, Consistent, Isolated, Durable) trait of the traditional databases in exchange with keeping a weaker BASE (Basic Availability, Soft state, Eventual consistency) feature. Remains to be seen which of the multiple solutions created since the official appearance of the NoSQL concept (which was defined in 1998 and reintroduced in 2009, around which moment several NoSQL solutions emerged; at the present moment there are known over 120 such solutions) are really delivering on these promises of higher performance (although several of them are already used with very good results).
---
paper_title: Data Mining Concepts and Techniques

paper_content:

Understand the need for analyses of large, complex, information-rich data sets. Identify the goals and primary tasks of the data-mining process. Describe the roots of data-mining technology. Recognize the iterative character of a data-mining process and specify its basic steps. Explain the influence of data quality on a data-mining process. Establish the relation between data warehousing and data mining. Data mining is an iterative process within which progress is defined by discovery, through either automatic or manual methods. Data mining is most useful in an exploratory analysis scenario in which there are no predetermined notions about what will constitute an "interesting" outcome. Data mining is the search for new, valuable, and nontrivial information in large volumes of data. It is a cooperative effort of humans and computers. Best results are achieved by balancing the knowledge of human experts in describing problems and goals with the search capabilities of computers. In practice, the two primary goals of data mining tend to be prediction and description. Prediction involves using some variables or fields in the data set to predict unknown or future values of other variables of interest. Description, on the other hand, focuses on finding patterns describing the data that can be interpreted by humans. Therefore, it is possible to put data-mining activities into one of two categories: Predictive data mining, which produces the model of the system described by the given data set, or Descriptive data mining, which produces new, nontrivial information based on the available data set.
---
paper_title: Serving Large-scale Batch Computed Data with Project Voldemort

paper_content:

Current serving systems lack the ability to bulk load massive immutable data sets without affecting serving performance. The performance degradation is largely due to index creation and modification as CPU and memory resources are shared with request serving. We have extended Project Voldemort, a general-purpose distributed storage and serving system inspired by Amazon's Dynamo, to support bulk loading terabytes of read-only data. This extension constructs the index offline, by leveraging the fault tolerance and parallelism of Hadoop. Compared to MySQL, our compact storage format and data deployment pipeline scales to twice the request throughput while maintaining sub 5 ms median latency. At LinkedIn, the largest professional social network, this system has been running in production for more than 2 years and serves many of the data-intensive social features on the site.
---
paper_title: Dynamo: amazon's highly available key-value store

paper_content:

Reliability at massive scale is one of the biggest challenges we face at Amazon.com, one of the largest e-commerce operations in the world; even the slightest outage has significant financial consequences and impacts customer trust. The Amazon.com platform, which provides services for many web sites worldwide, is implemented on top of an infrastructure of tens of thousands of servers and network components located in many datacenters around the world. At this scale, small and large components fail continuously and the way persistent state is managed in the face of these failures drives the reliability and scalability of the software systems. This paper presents the design and implementation of Dynamo, a highly available key-value storage system that some of Amazon's core services use to provide an "always-on" experience. To achieve this level of availability, Dynamo sacrifices consistency under certain failure scenarios. It makes extensive use of object versioning and application-assisted conflict resolution in a manner that provides a novel interface for developers to use.
---
paper_title: Bigtable: A Distributed Storage System for Structured Data

paper_content:

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.
---
paper_title: Hadoop-HBase for large-scale data

paper_content:

Today we are inundated with digital data. Yet we are very poor in managing and processing it. It is becoming increasingly difficult to store and analyze data efficiently and economically via conventional database management tools. Not only that, type of data, appearing in the databases, are also changing. Now a day, binary large objects are a standard integral part of any database. Researchers, all over the globe, are baffling with analysis of these ultra large databases. Apache HBase is one such attempt. HBase is a noSQL distributed database developed on top of Hadoop Distributed File System (HDFS). In this paper, we present an evaluation of hybrid architecture where HDFS contains the non-textual data like images and location of such data is stored in HBase. This hybrid architecture enables faster search and retrieval of the data which is a growing need in any organization who are flooded with data. The paper aims at evaluating the performance of random reads and random writes of data storage location information to HBase and retrieving and storing data in HDFS respectively. We also present a comparative study of HBase-HDFS architecture with MySQL-HDFS architecture.
---
paper_title: A Decentralized Architecture for SPARQL Query Processing and RDF Sharing: A Position Paper

paper_content:

In this position paper, we propose a decentralized architecture for publishing RDF. Unlike traditional client-server architectures, we propose a peer-to-peer (P2P) architecture for SPARQL query processing and RDF knowledge sharing. We present a general architecture and its key problems. Further, we discuss its potential with respect to solving the availability and performance issues that for a long time have been prevailing on existing infrastructures.
---
paper_title: Cypher: An Evolving Query Language for Property Graphs

paper_content:

The Cypher property graph query language is an evolving language, originally designed and implemented as part of the Neo4j graph database, and it is currently used by several commercial database products and researchers. We describe Cypher 9, which is the first version of the language governed by the openCypher Implementers Group. We first introduce the language by example, and describe its uses in industry. We then provide a formal semantic definition of the core read-query features of Cypher, including its variant of the property graph data model, and its ASCII Art graph pattern matching mechanism for expressing subgraphs of interest to an application. We compare the features of Cypher to other property graph query languages, and describe extensions, at an advanced stage of development, which will form part of Cypher 10, turning the language into a compositional language which supports graph projections and multiple named graphs.
---
paper_title: Resilience engineering: learning to embrace failure

paper_content:

A discussion with Jesse Robbins, Kripa Krishnan, John Allspaw, and Tom Limoncelli.
---
paper_title: A certain freedom: thoughts on the CAP theorem

paper_content:

At PODC 2000, the CAP theorem received its first broad audience. Surprisingly for an impossibility result, one important effect has been to free designers to explore a wider range of distributed systems. Designers of wide-area systems, in which network partitions are considered inevitable, know they cannot have both availability and consistency, and thus can now justify weaker consistency. The rise of the "NoSQL" movement ("Not Only SQL") is an expression of this freedom. The choices of how and when to weaken consistency are often the defining characteristics of these systems, with new variations appearing every year. We review a variety of interesting places in the "CAP Space" as a way to illuminate these issues and their consequences. For example, automatic teller machines (ATMs), which predate the CAP theorem, surprisingly choose availability with weak consistency but with bounded risk. Finally, I explore a few of the options to try to "work around" the impossible. The most basic is the use of commutative operations, which make it easy to restore consistency after a partition heals. However, even many commutative operations have non-commutative exceptions in practice, which means that the exceptions may be incorrect or late. Adding the concept of "delayed exceptions" allows more operations to be considered commutative and simplifies eventual consistency during a partition. Furthermore, we can think of delayed exception handling as "compensation" - we execute a compensating transaction that restores consistency. Delayed exception handling with compensation appears to be what most real wide-area systems do - inconsistency due to limited communication is treated as an exception and some exceptional action, such as monetary compensation or even legal action, is used to fix it. This approach to wide-area systems puts the emphasis on audit trails and recovery rather than prevention, and implies that we should expand and formalize the role of compensation in the design of complex systems
---
paper_title: CAP twelve years later: How the "rules" have changed

paper_content:

The CAP theorem asserts that any networked shared-data system can have only two of three desirable properties. However, by explicitly handling partitions, designers can optimize consistency and availability, thereby achieving some trade-off of all three. The featured Web extra is a podcast from Software Engineering Radio, in which the host interviews Dwight Merriman about the emerging NoSQL movement, the three types of nonrelational data stores, Brewer's CAP theorem, and much more.
---
paper_title: C-Store: A Column-oriented DBMS

paper_content:

This paper presents the design of a read-optimized relational DBMS that contrasts sharply with most current systems, which are write-optimized. Among the many differences in its design are: storage of data by column rather than by row, careful coding and packing of objects into storage including main memory during query processing, storing an overlapping collection of column-oriented projections, rather than the current fare of tables and indexes, a non-traditional implementation of transactions which includes high availability and snapshot isolation for read-only transactions, and the extensive use of bitmap indexes to complement B-tree structures.We present preliminary performance data on a subset of TPC-H and show that the system we are building, C-Store, is substantially faster than popular commercial products. Hence, the architecture looks very encouraging.
---
paper_title: Consistency Tradeoffs in Modern Distributed Database System Design: CAP is Only Part of the Story

paper_content:

The CAP theorem's impact on modern distributed database system design is more limited than is often perceived. Another tradeoff-between consistency and latency -has had a more direct influence on several well-known DDBSs. A proposed new formulation, PACELC, unifies this tradeoff with CAP.
---
paper_title: Perspectives on the CAP Theorem

paper_content:

The CAP theorem is one example of a more general tradeoff between safety and liveness in unreliable systems. Viewing CAP in this context provides insight into the inherent tradeoffs and the manner in which they can be circumvented in practice.
---
paper_title: Database Consistency Models

paper_content:

A data store allows application processes to put and get data from a shared memory. In general, a data store cannot be modelled as a strictly sequential process. Applications observe non-sequential behaviours, called anomalies. The set of pos- sible behaviours, and conversely of possible anomalies, constitutes the consistency model of the data store.
---
paper_title: Consistency Models of NoSQL Databases

paper_content:

Internet has become so widespread that most popular websites are accessed by hundreds of millions of people on a daily basis. Monolithic architectures, which were frequently used in the past, were mostly composed of traditional relational database management systems, but quickly have become incapable of sustaining high data traffic very common these days. Meanwhile, NoSQL databases have emerged to provide some missing properties in relational databases like the schema-less design, horizontal scaling, and eventual consistency. This paper analyzes and compares the consistency model implementation on five popular NoSQL databases: Redis, Cassandra, MongoDB, Neo4j, and OrientDB. All of which offer at least eventual consistency, and some have the option of supporting strong consistency. However, imposing strong consistency will result in less availability when subject to network partition events.
---
paper_title: Eventually Consistent

paper_content:

Building reliable distributed systems at a worldwide scale demands trade-offs?between consistency and availability.
---
paper_title: Big data: The next frontier for innovation, competition, and productivity

paper_content:

The amount of data in our world has been exploding, and analyzing large data sets—so-called big data— will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.
---
paper_title: A CAPable distributed programming model

paper_content:

Developers of modern distributed systems continuously face the impossibility result proved by the CAP theorem. In a nutshell, the theorem states that a partition-tolerant system can either guarantee consistency or availability. Most distributed programming languages implicitly make the choice between consistency or availability in their designs and implementations. Concretely, distributed programming languages can be roughly divided into two categories. A first category of languages provide abstractions to implement the consistent parts of a distributed system. A second category of languages provide abstractions to implement the available parts of a distributed system. However, real-world distributed systems often require consistency for some parts while requiring availability for others. Programmers are therefore forced to implement the abstractions missing from their chosen distributed programming language themselves or rely on external libraries. In this paper we present a novel distributed programming model. This model introduces two object-oriented abstractions: consistents and availables. The former guarantees strong consistency by sacrificing availability. The latter guarantees availability, but only provides eventual consistency. Through these constructs programmers are able to implement the entirety of their distributed system within the same language. We present a prototypical implementation of the model as a TypeScript library called CAPtain.js. To showcase the usefulness of our approach we implement a non-trivial example application. Moreover, we highlight both the functional as well as the performance characteristics of both language abstractions.
---
paper_title: Scalable SQL and NoSQL data stores

paper_content:

In this paper, we examine a number of SQL and socalled "NoSQL" data stores designed to scale simple OLTP-style application loads over many servers. Originally motivated by Web 2.0 applications, these systems are designed to scale to thousands or millions of users doing updates as well as reads, in contrast to traditional DBMSs and data warehouses. We contrast the new systems on their data model, consistency mechanisms, storage mechanisms, durability guarantees, availability, query support, and other dimensions. These systems typically sacrifice some of these dimensions, e.g. database-wide transaction consistency, in order to achieve others, e.g. higher availability and scalability.
---
paper_title: A Polyglot Persistence approach for E-Commerce business model

paper_content:

E-Commerce is one of the fastest growing business models in this era of Big data. Advancements in infrastructure technologies and data processing techniques are making E-Commerce a flourishing business. Demand of rapid scalability and availability is proving relational database as a legacy system but due to its evolved state and ease of implementation most of the business models are still not shifting from relational to No SQL. But need of scalability and data processing for unstructured data is not handled properly by relation databases. Today's business models such as E Commerce need a shift from relational to the principles of No SQL. Applying one No SQL for all the modules of E-Commerce is not feasible as every module has different requirements. Some modules require consistency others require availability. There is a need of multiple No SQL database. Every No SQL has its pros and cons. Applying the concept of Polyglot Persistence (by Neal Ford, 2006), where different modules can have their own different data processing mechanism, can solve the problem of scalability and processing unstructured data as well as maintaining legacy processing.
---
paper_title: Transactional Processing for Polyglot Persistence

paper_content:

NoSQL data stores have emerged in last years as a solution to provide scalability and flexibility in data modelling for operational databases. These data stores have proven that they are better suited for some kinds of problems than relational databases. In order to scale, they relaxed the properties provided by relational databases, mainly transactions. However, transactional semantics is still needed by most applications. In this paper we describe how CoherentPaaS provides scalable holistic transactions across SQL and NoSQL data stores such as document-oriented data stores, key-value data stores and graph databases.
---
paper_title: Survey on NoSQL database

paper_content:

With the development of the Internet and cloud computing, there need databases to be able to store and process big data effectively, demand for high-performance when reading and writing, so the traditional relational database is facing many new challenges. Especially in large scale and high-concurrency applications, such as search engines and SNS, using the relational database to store and query dynamic user data has appeared to be inadequate. In this case, NoSQL database created. This paper describes the background, basic characteristics, data model of NoSQL. In addition, this paper classifies NoSQL databases according to the CAP theorem. Finally, the mainstream NoSQL databases are separately described in detail, and extract some properties to help enterprises to choose NoSQL.
---
paper_title: Application of polyglot persistence to enhance performance of the energy data management systems

paper_content:

Energy Data Management (EDM) Utilities cater to the needs of energy data management, dissemination and analytics. Data management applies management of data collected from smart grids and smart meters, meter data analytics. The use of one database for different kinds of requirements may lead to non-performing solutions; storing energy generation, distribution and consumption data from smart grids, storing transactional data, caching session information, storing customer profile information, traversing graph of energy dissemination data and storage and retrieval of time series data from smart meters are different problems. Polyglot persistence refers to usage of different data storage technologies by taking into consideration the different ways in which multiple applications use the data. Our proposed approach gives a solution that allows collective access to a combination of databases by taking advantage of the concept known as polyglot persistence. A prototype of the above concept applied to Energy Data Management Systems (EDMS) is made as a proof of concept and results are shared in this paper.
---
paper_title: A novel polyglot data mapper for an E-Commerce business model

paper_content:

In the current scenario E-Commerce sites are replete with content. However, they are facing major challenges in areas of scalability and availability. Entity-attribute data designs are very complex and provide less flexibility to the data layer of the application. Relational databases are better for transactions and ACID guarantees but they prove to be a limiting factor in providing linear scalability for large volumes of data. Schema less No SQL stores are better when scalability is a requirement. Every No SQL data store has its own pros and cons; hence simply replacing legacy RDBMS systems with a single No SQL does not suffice. Rather, one needs to have a combination of different data stores for different modules of a business model. However, addition of multiple data stores in the application brings new challenges and increases complexity of maintaining the application. In this paper, we present a hybrid data store architecture using a polyglot data mapper for E-Commerce aiming to achieve better scalability whilst ensuring simplistic maintenance of the application infrastructure.
---
paper_title: NextGen data persistence pattern in healthcare: Polyglot persistence

paper_content:

Databases are the “Backbone” of Hospital Information Systems. The storage of transactional data, session information, Multi Modal data storage and retrieval, traversing graph of treatments the patient has undergone and specialist doctors they visited are essentially different problems. Different databases are designed to overcome different storage problems. Using a single database engine for all of the requirements usually leads to non-performant solutions [1]. Many online healthcare enterprises tend to use the same database engine to store business transactions, session management data, and for other storage needs such as reporting, BI, data warehousing, Multi Modal data or logging information, as is typical case with online hospital and clinical information management systems. The user sessions, patient registration details, recommendations by doctors, appointments registered, billing information, maintenance or inventory order data do not need the same properties of availability, consistency, or backup requirements. Does session management storage need the same rigorous backup/recovery strategy as the multi modal medical data backup or patient medical data? One-database-fits-all is no more the de facto choice for web applications [4]. The need of situation is to adopt for notion of polyglot persistence. Basic idea behind polyglot persistence is to use specialized databases (both NoSQL and Relational in combination) depending upon the performance parameters required by different healthcare related information within the same healthcare web application. Our proposed architecture provides a solution to augment the access to such disparate data stores using the concept of polyglot persistence. In this paper, we provide a prototype implementation of polyglot persistence applied to healthcare Information Systems as a proof-of-concept.
---
paper_title: ExSchema: Discovering and Maintaining Schemas from Polyglot Persistence Applications

paper_content:

The use of scalable and heterogeneous data stores within a single system is gradually becoming a common practice in application development. Modern applications tend to rely on a polyglot approach to persistence, where traditional databases, non-relational data stores, and scalable systems associated to the emerging New SQL movement, are used simultaneously. However, considering that a large number of these systems rely on schema-less data models, developers in charge of maintaining applications that depend on these data stores need to manually analyze the application source code, in order to discover their data schemas. To help overcome this situation, in this paper we demonstrate ExSchema, a tool for the automatic discovery of external data schemas, directly from the source code of polyglot persistence applications. ExSchema is available as an Eclipse plug in, and can be easily included within existing development environments. Our tool can also be integrated with Git repositories, in order to analyze the evolution of the schemas used by an application over a period of time. In particular, this paper presents a demonstration scenario where the schemas from a system that relies on multiple document, graph, relational and column-family data stores, are discovered from the application source code.
---
paper_title: Managing Data in Healthcare Information Systems: Many Models, One Solution

paper_content:

Because healthcare data comes from multiple, vastly different sources, databases must adopt a range of models to process and store it. A polyglot-persistent framework combines relational, graph, and document data models to accommodate information variety.
---
paper_title: Scalable data management: NoSQL data stores in research and practice

paper_content:

The unprecedented scale at which data is consumed and generated today has shown a large demand for scalable data management and given rise to non-relational, distributed “NoSQL” database systems. Two central problems triggered this process: 1) vast amounts of user-generated content in modern applications and the resulting requests loads and data volumes 2) the desire of the developer community to employ problem-specific data models for storage and querying. To address these needs, various data stores have been developed by both industry and research, arguing that the era of one-size-fits-all database systems is over. The heterogeneity and sheer amount of these systems - now commonly referred to as NoSQL data stores - make it increasingly difficult to select the most appropriate system for a given application. Therefore, these systems are frequently combined in polyglot persistence architectures to leverage each system in its respective sweet spot. This tutorial gives an in-depth survey of the most relevant NoSQL databases to provide comparative classification and highlight open challenges. To this end, we analyze the approach of each system to derive its scalability, availability, consistency, data modeling and querying characteristics. We present how each system's design is governed by a central set of trade-offs over irreconcilable system properties. We then cover recent research results in distributed data management to illustrate that some shortcomings of NoSQL systems could already be solved in practice, whereas other NoSQL data management problems pose interesting and unsolved research challenges.
---
paper_title: Standardization of NoSQL Database Languages

paper_content:

NoSQL database systems have been becoming more and more popular and accepted by a database users thus their rapid development is nowadays observed. Because of this fact, modern database engines and their categories in the form of the Venn diagram are mentioned in the paper. Besides, the possibilities of using declarative languages that are modeled on SQL - the language for relational databases – in NoSQL, are presented. For this purpose selected NoSQL technologies are given in more details and their query languages are described. Moreover, the NoSQL language commands’ equivalents of SQL standard are provided in this document.
---
paper_title: Towards a Scalable and Unified REST API for Cloud Data Stores.

paper_content:

In the last years, many database-as-a-service (DBaaS) systems have started to offer their functionalities through REST APIs. Examples are record stores like DynamoDB and Azure Tables, object stores such as Amazon S3 as well as many NoSQL database systems, for instance Riak, CouchDB and ElasticSearch. Yet today, there has been no systematic effort on deriving a unified REST interface which takes into account the different data models, schemas, consistency concepts, transactions, access-control mechanisms and query languages to expose cloud data stores through a common interface without restricting their functionality. This work motivates the design of such a REST API as well as the challenges of providing it in an extensible, scalable and highly-available fashion. To this end, we propose the REST middleware ORESTES that consists of an independently scalable tier of HTTP servers that map the unified REST API to aggregate-oriented (NoSQL) data stores. It extracts a wide range of DBaaS concerns (e.g. schema management and access control) and provides them in a modular, database-independent fashion at the middleware-level. To tackle the latency problem of cloud-based web applications we introduce the Bloom filter-bounded staleness cache consistency algorithm. It leverages the global web caching infrastructure for geo-replication to allow consistent low latency reads. We furthermore show the first steps towards a Polyglot Persistence Mediator that exploits the decoupling of the REST API from the data store to route data and operations based on SLAs.
---
paper_title: UDBMS: Road to Unification for Multi-model Data Management

paper_content:

One of the greatest challenges in big data management is the “Variety” of the data. The data may be presented in various types and formats: structured, semi-structured and unstructured. For instance, data can be modeled as relational, key-value, and graph models. Having a single data platform for managing both well-structured data and NoSQL data is beneficial to users; this approach reduces significantly integration, migration, development, maintenance, and operational issues. Therefore, a challenging research work is how to develop an efficient consolidated single data management platform covering both NoSQL and relational data to reduce integration issues, simplify operations, and eliminate migration issues. In this paper, we envision novel principles and technologies to handle multiple models of data in one unified database system, including model-agnostic storage, unified query processing and indexes, in-memory structures and multi-model transactions. We discuss our visions as well as present research challenges that we need to address.
---
paper_title: Reference Architecture and Classification of Technologies, Products and Services for Big Data Systems

paper_content:

Many business cases exploiting big data have been realised in recent years; Twitter, LinkedIn, and Facebook are examples of companies in the social networking domain. Other big data use cases have focused on capturing of value from streaming of movies (Netflix), monitoring of network traffic, or improvement of processes in the manufacturing industry. Also, implementation architectures of the use cases have been published. However, conceptual work integrating the approaches into one coherent reference architecture has been limited. The contribution of this paper is technology independent reference architecture for big data systems, which is based on analysis of published implementation architectures of big data use cases. An additional contribution is classification of related implementation technologies and products/services, which is based on analysis of the published use cases and survey of related work. The reference architecture and associated classification are aimed for facilitating architecture design and selection of technologies or commercial solutions, when constructing big data systems.
---
paper_title: Data transformation as a means towards dynamic data storage and polyglot persistence

paper_content:

Legacy applications have been built around the concept of storing their data in one relational data store. However, with the current differentiation in data store technologies as a consequence of the NoSQL paradigm, new and possibly more performant storage solutions are available to all applications. The concept of dynamic storage makes sure that application data are always stored in the most optimal data store at a given time to increase application performance. Additionally, polyglot persistence aims to push this performance even further by storing each different data type of an application in the data store technology best suited for it. To get legacy applications into dynamic storage and polyglot persistence, schema and data transformations between data store technologies are needed. This usually infers application redesigns as well to support the new data stores. This paper proposes such a transformation approach through a canonical model. It is based on the Lambda architecture to ensure no application downtime is needed during the transformation process, and after the transformation, the application can continue to query in the original query language, thus requiring no application code changes.
---
paper_title: Perspectives on the CAP Theorem

paper_content:

The CAP theorem is one example of a more general tradeoff between safety and liveness in unreliable systems. Viewing CAP in this context provides insight into the inherent tradeoffs and the manner in which they can be circumvented in practice.
---

Title: A Review of Polyglot Persistence in the Big Data World

Section 1: Introduction
Description 1: Introduce the concept of big data and the challenges associated with its management, especially the need for diverse data management approaches beyond traditional SQL databases.

Section 2: Technical and Business Perspective: A Brief Look at SQL and NoSQL Stores in the Big Data Landscape
Description 2: Discuss the diverse nature of data in big data environments and the limitations of SQL databases, highlighting the rise of NoSQL databases.

Section 3: (Relational) SQL Databases
Description 3: Detail the characteristics and structure of relational SQL databases, including their historical development and limitations in the context of big data.

Section 4: NoSQL Data Stores
Description 4: Examine the emergence and characteristics of NoSQL databases, including key concepts and their departure from traditional SQL methods.

Section 5: Types of NoSQL Data Stores
Description 5: Describe the four primary types of NoSQL databases: key-value stores, columnar stores, document stores, and graph stores, along with their unique features and use cases.

Section 6: CAP Theorem of Big Data Era-Relationship of Polyglot Persistence with CAP Theorem
Description 6: Explain CAP Theorem and its relevance to NoSQL databases, highlighting how it influences database consistency, availability, and partition tolerance.

Section 7: BASE Descending from the CAP Theorem
Description 7: Discuss the BASE model as an alternative to ACID in the context of NoSQL databases and its implications for big data systems.

Section 8: CAP Compromises
Description 8: Explore the trade-offs between consistency, availability, and partition tolerance in NoSQL databases, and the implications of the PACELC extension to the CAP theorem.

Section 9: Varying Consistency Levels, NoSQL, SQL (Strong Consistency) and Polyglot Persistence
Description 9: Review different consistency models offered by NoSQL databases and their strategic applications in polyglot persistence.

Section 10: No One Cure-Them-All Database Resulting in Polyglot Persistence
Description 10: Discuss the necessity of polyglot persistence due to the limitations of both SQL and NoSQL databases in handling all types of big data needs.

Section 11: Evolution of Polyglot Persistence
Description 11: Trace the development and adoption of polyglot persistence, detailing how different database types are used within single systems.

Section 12: The Simplest Form of Polyglot Persistence
Description 12: Provide examples and a basic explanation of polyglot persistence implementations in real-world applications.

Section 13: Choosing NoSQL Databases for Different Parts of the System
Description 13: Offer guidelines on selecting appropriate NoSQL databases for various components of a big data system based on specific requirements.

Section 14: Polyglot Persistence Implementation Examples
Description 14: Provide examples of polyglot persistence implementations and discuss how they enhance system performance and flexibility.

Section 15: Approaches to Achieve Polyglot Persistence
Description 15: Present various methods to implement polyglot persistence, including domain-oriented solutions, query language-based solutions, and general frameworks.

Section 16: Polyglot Persistence in Big Data Architectures
Description 16: Explore how polyglot persistence is integrated into big data architectures like Lambda and Kappa, and discuss their effectiveness and challenges.

Section 17: Discussion
Description 17: Summarize the challenges and future directions for polyglot persistence, including data integration and the need for a universal query language.

Section 18: Conclusions
Description 18: Conclude with reflections on the evolving concepts in big data management, the role of polyglot persistence, and its impact on technology and methodologies.