« Hive - A Petabyte Scale Data Warehouse using Hadoop | Main |
Paper: Graph Databases and the Future of Large-Scale Knowledge
Management »  Dealing with multi-partition transactions in a
distributed KV solution  I've been getting asked about this a
lot lately so I figured I'd just blog about it. Products like
WebSphere eXtreme Scale work by taking a dataset, partitioning
it using a key and then assigning those partitions to a number
of JVMs. Each partition usually has a primary and a replica.
These 'shards' are assigned to JVMs. A transactional application
typically interacts with the data on a single partition at a
time. This means the transaction is executed in a single JVM. A
server box will be able to do M of those transactions per second
and it scales because N boxes does MN (M multiplied by N)
transactions per second. Increase N, you get more transactions
per second. Availability is very good because a transaction only
depends on 1 of the N servers that are currently online. Any of
the other (N-1) serve