

Big Data Terminology

September 12, 2016, November 03, 2016 | Comments

category: BigData

Accumulo - A computer software project that developed a sorted, distributed key/value store based on the BigTable technology from Google. It is a system built on top of Apache Hadoop, Apache ZooKeeper, and Apache Thrift. Written in Java, Accumulo has cell-level access labels and server-side programming mechanisms.

Amazon EC2 - Amazon Elastic Compute Cloud, is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers.

Amazon EC2 Container Service (ECS) - A highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.

Amazon S3 - Amazon Simple Storage Service, provides developers and IT teams with secure, durable, highly-scalable cloud storage.


AWS - Amazon Web Services, a suite of cloud-computing services that make up an on-demand computing platform.

Cassandra - A free and open-source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. It offers robust support for clusters spanning multiple datacenters,[1] with asynchronous masterless replication allowing low latency operations for all clients.


DAG - Directed Acyclic Graph. When a SparkContext is created, it is submitted to DAGScheduler. A stage is comprised of tasks based on partitions of the input data. The DAG scheduler pipelines operators together. The Stages are passed on to the Task Scheduler. The task scheduler launches tasks via cluster manager. And finally the Worker executes the tasks on the Slave.


Flink - An open source platform for distributed stream and batch data processing. The core of it is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.



HBase - A column-oriented database management system that runs on top of HDFS.


Hive - A data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. It gives an SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop.

Kafka - A distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system.



MapReduce - A programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.


MongoDB - A cross-platform, open-source database that uses a document-oriented data model, rather than a traditional table-based relational database structure. This type of database structure is designed to make the integration of structured and unstructured data in certain types of applications easier and faster.

Oozie - A Java Web application used to schedule Apache Hadoop jobs. It is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp) as well as system specific jobs (such as Java programs and shell scripts).




REST - REpresentational State Transfer, an architectural style, and an approach to communications that is often used in the development of Web services.

Shark - Also known as SQL on Spark, is a large-scale data warehouse system for Spark designed to be compatible with Apache Hive. Shark supports Hive’s query language, metastore, serialization formats, and user-defined functions, providing seamless integration with existing Hive deployments and a familiar, more powerful option for new ones. It has been subsumed by Spark SQL.



Sqoop - A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.

Storm - A distributed real-time computation system for processing large volumes of high-velocity data. It is extremely fast, with the ability to process over a million records per second per node on a cluster of modest size.

Tez - An extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop. It improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce’s ability to scale to petabytes of data.



ZooKeeper - A centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.