Ministry of Innovation —



But what about the files that each client reads and writes? The Isilon cluster presents all of its storage in a big single namespace, and so what happens if users 1, 2, and 3 all want to read and write to the same files, even though they're connecting to three different nodes? Again, the answer comes from the highly distributed nature of OneFS and from all the nodes acting in concert. Remember that the cluster stripes files based on the N+M protection level (or mirrors them, if that's what it's supposed to do). Exactly how a file is striped, and to exactly what nodes it gets striped, is controlled by the node that first receives that particular file from a client.

The node that the client corresponds with is the one that controls the striping for the write, and the reconstruction for the read

In step 1, a user writes a PDF file to our ten-node cluster, which we've configured to give 9+1 protection to our files. In step 2, the node to which the user copied the file starts dividing the file up into 128k blocks, and since this is an example, we're going to say that our file is a fortuitous 3,456MB in size, which chops up nicely into twenty-seven 128KB chunks. The node does the chopping, then communicates over the back-end Infiniband IP network with the other cluster nodes and finds out who's not busy at the moment and who's got space available. Then in step 3, the node spits out the 128KB chunks over the back-end network to the other nodes in three ten-element stripes, each stripe element landing on a different disk.

The process for reading back files is the same, but in reverse (and I didn't make any diagrams for that). A client asks the node to which it's it's connected for whatever file it wants. The node checks its copy of the file system journal to see where the pieces that make up that particular file have been distributed on the cluster, and then contacts those other nodes over the back-end network. It gets those nodes to send it the file's chunks, and then reassembles them in its cache. Finally, it sends the file back out to the client.

There's a significant amount of communication going back and forth between all of the nodes all the time—every single operation requires the nodes to communicate, and beyond that, the nodes are each continually attempting to keep their percentage of free vs. used space roughly equal, and so there's a low-priority restriping task constantly running to keep things balanced. This is the main reason for using Infiniband to hook the nodes together—Infiniband's extremely low latency means that the back-end network can handle a tremendous amount of IO without becoming saturated.

Scaling out

The last thing to touch on is cluster expansion. Isilon's biggest marketing point, and one that there are lots and lots of papers about, is its ability to adapt to changing workloads by adding different types of nodes into the mix. Like I mentioned way back at the beginning of the article, there are nodes that contribute CPU and network throughput; there are nodes that are big SATA containers that don't add much CPU but add lots of capacity; there are nodes with some solid-state disks and some regular disks that add tons of IOPS, and there are middle-of-the-road nodes that add a mix of CPU and fast disk.

That's great, but since the whole purpose of the cluster is to present one big glob of storage under a single name, what happens when the storage in the cluster isn't homogenous in nature? If you've got a mix of SSDs and near-line SATA and Fibre Channel disks, how does the cluster decide what goes on the SSDs and what goes on the SATA disks?

Most of the big storage solutions these days have their method of auto-tiering—that is, letting data (either files or blocks, depending on the method) move automatically between faster or slower disks based on how hard the data are being used. Isilon calls their auto-tiering feature SmartPools (Isilon loves its CamelCase words). In a system with different types of nodes—maybe you've got some big near-line storage buckets, maybe a few nodes with SSDs, and a few Fibre Channel nodes, too—the system lets the administrator define performance policies for each type of node, and then the system watches to see how files and directories are being accessed and moves them around accordingly. The system administrator can also manually specify that certain files or directories should be optimized for fast access to bump them up, if needed. The rule-set governing what goes where and why can be made complex or simple, depending on the requirements, but the convenience factor of being able to pop in a few performance nodes in order to pep up a laggy system is extremely high.

It's been a long road, getting from there to here

And so we circle all the way back to the opening premise—with all this being said, what makes Isilon a fit for a company like Apple to use for hosting something like the iTunes back-end? Isilon fits this task, which involves ingesting and regurgitating lots of big files, because it takes those big files and stripes them out across lots of disks in lots of nodes, and each node has its own CPU and global cache and network resources to service IO. Further, the cluster can be expanded on the fly, in the middle of production, without downtime, as new files come in or new workloads develop. Further still, those expansions bring with them not just additional disk but additional CPU, cache, and networking, and so adding nodes can make the entire system not just bigger, but faster, even for existing shares.

From a technology standpoint, Isilon's take on IO distribution and slicing at the scale they're presenting seems to be unique, and it's something I've been fascinated with for a long time. In some ways, it kind of resembles Drobo's BeyondRAID method of pooling and storage presentation, though the similarities are superficial at best—OneFS is a node-based NAS operating system that distributes files in stripes and mirrors across nodes, scales to north of 10,000 TB, and can handle hundreds of thousands of concurrent IOPS and tens of gigabits of throughput per second at scale; BeyondRAID is a disk-based method of pooling that distributes files in stripes and mirrors across a few disks in a single system, and can handle a much smaller workload.

Isilon NAS clusters are big, though, and that's why I wanted to put this article together—big stuff is cool. There are lots of neat technologies working in data centers around the world in monstrous racks, and I very much enjoy learning about all of them in depth, no matter who makes them or what they're for. It's a trait that I think a lot of geeks share.

You must to comment.