In-Memory Compute Grid… Explained.

Dmitriy Setrakyan provided an excellent explanation for In-Memory Data Grid (IMDG) in his blog

I will try to provide a similar description for In-Memory Compute Grid (IMCG).

PDF version of this article is available.

IMCG – In-Memory Compute Grid

One of the main ideas Dmitriy put forward is the importance of integration between in-memory storage (IMDG) and in-memory processing (IMCG) to be able to build truly scalable applications. Yet – the IMCG and its implementations are seen less frequently than IMDG mainly due to the historical reason described below.

Most vendors to this day concentrate first on storage technology (IMDG, NoSQL, or NewSQL variety). Once the storage product is built – adding any type of non-rudimentary IMCG capability on top of it becomes increasingly difficult, if not impossible overall (we'll see why it is so below). Thus, generally, IMCG capabilities are more fundamental to the overall product and therefore have to be built first or together to be used at the core of the storage side.

It should be no surprise, by the way, that GridGain and Hadoop are still the only products on the market that successfully combine both storage and processing in one product (although very differently), while there are dozens of storage-only projects available (and probably hundreds if you count every NoSQL attempt on GitHub).

Core Concepts

The easiest way to understand IMCGs is through a comparison to IMDGs. While IMDGs focus on distributed in-memory storage and management of large data sets by partitioning this data across available computers in the grid, IMCG concentrate on efficiently executing algorithms (i.e. user's code or instructions) across the same set of computers on the same grid. And that's all there's to it: IMDG is all about storing and managing data in-memory, and IMCG is all about processing and computing across the same data.

When seen from this vantage point – it is pretty clear why tight integration between IMDG and IMCG is so important: they are practically two sides of the same coin – storage and processing, that both coalesce around your data.

Most of the functionality in any IMCG can be split into four individual groups:

  1. Distributed Deployment & Provisioning
  2. Distributed Resources Management
  3. Distributed Execution Models (a.k.a. IMCG Breadth)
  4. Distributed Execution Services (a.k.a. IMCG Depth)

1. Distributed Deployment & Provisioning

Historically deployment and provisioning of the user's code onto the grid for execution was one of the core reasons why grid computing in general was considered awkward and cumbersome at best, and downright unusable at worst. From the early products like Globus, Grid Engine, DataSynapse, Platform Computing, and such, to today's Hadoop and most of the NoSQL projects – deploying and re-deploying your changes is a manual step that involves rebuilding all of your libraries, copying them everywhere, and restarting your services. Some systems will do copying & restarting for you (Hadoop) and some will require you to do it manually via some UI-based crutch.

This problem is naturally exacerbated by the fact that IMCGs are a distributed technology to begin with and are routinely used on topologies consisting of dozens if not hundreds of computers. Stopping services, redeploying libraries and re-starting services during developing, CI testing and staging in such topologies becomes a major issue.

GridGain is the first IMCG that simplifies this issue by providing "zero deployment" capabilities. With "zero depoloyment" all necessary JVM classes and resources are loaded on demand. Further, GridGain provides three different modes of peer-to-peer deployment supporting the most complex deployment environments like custom class loaders, WAR/EAR files, etc.

Zero deployment technology enables users to simply bring default GridGain nodes online with these nodes then immediately becoming part of the data and compute grid topology that can store any user objects or perform any user tasks without any need for explicit deployment of user’s classes or resources.

2. Distributed Resources Management

Resource management in distributed systems usually refers to the ability to manage physical devices such as computers, networks, and storage as well as software components like JVM, runtimes and OSes. Specifics of that obviously differ based on whether or not the IMCG is deployed on some kind of managed infrastructure like AWS, how it is DevOps managed, etc.

One of the most important resource management functions of any IMCG is automatic discovery and maintaining consistent topology (i.e. the set of compute nodes). Automatic discovery allows the user to add and remove compute nodes from the IMCG topology at runtime while maintaining zero downtime for the tasks running on the IMCG. Consistent topology ensures that any topology changes (nodes failing and leaving, or new nodes joining) viewed by all compute nodes in the same order and consistently.

GridGain provides the most sophisticated discovery system among any IMCG. Pluggable and user-defined Discovery SPI is at the core of GridGain's ability to provide fully automatic and consistent discovery functionality for GridGain nodes. GridGain is shipped with several out-of-the-box implementations including IP-multicast- and TCP/IP-based implementations with direct support for AWS S3 and Zookeeper.

3. Distributed Execution Models (a.k.a IMCG Breadth)

Support for different distributed execution models is what makes IMCG a compute framework. For clarity let's draw a clear distinction between an execution model (such as MapReduce) and the particular algorithms that can be implemented using this model (i.e. Distributed Search): there is a finite set of execution models but practically an infinite set of possible algorithms.

Generally, the goal of any IMCG (as well as of any compute framework in general) is to support as many different execution models as possible, providing the end-user with the widest set of options on how a particular algorithm can be implemented and ultimately executed in the distributed environment. That's why we often call it IMCG Breadth.

GridGain's IMCG, for a example, provides direct support for the following execution models:

3. Distributed Execution Services (a.k.a IMCG Depth)

In many respects the distributed execution services is the "meat" around proverbial execution models' "bones". Execution services refer to many dozens of deep IMCG features that support various execution strategies and models including services such as distributed failover, load balancing, collision resolution, etc. – hence the moniker of IMCG Depths.

Many such features are shared between different IMCGs and general compute frameworks – but some are unique to a particular product. Here is a short list of some of the key execution services provided by GridGain's IMCG:


The following examples demonstrate a typical stateless computation task of Pi-number calculation on the grid (written in Scala – but can be easily done in Java or Groovy or Clojure as well). This example shows how tremendously simple the implementation can be with GridGain – literally just a dozen lines of code.

Note that this is a full source code – copy'n'paste it, compile it and run it. Note also that it works on one node – and just as well on a thousand nodes in the grid or cloud with no code change – just linearly faster. What is even more interesting is that this application automatically includes all these execution services:

Scala code:

import org.gridgain.scalar._
import scalar._
import scala.math._

object ScalarPiCalculationExample {
    private val N = 10000

    def main(args: Array[String]) {
        scalar {
            println("Pi estimate: " +
                grid$.spreadReduce(for (i <- 0 until grid$.size()) yield () => calcPi(i * N))(_.sum))

    def calcPi(start: Int): Double =
        // Nilakantha algorithm.
        ((max(start, 1) until (start + N)) map 
            (i => 4.0 * (2 * (i % 2) - 1) / (2 * i) / (2 * i + 1) / (2 * i + 2)))
            .sum + (if (start == 0) 3 else 0)
© 2008 SYS-CON Media