Article Index

Dollars per Jigawatt

One of the fun measures that people like to look at is dollars per GFLOPS (Giga Floating Point Operations per second). I'm going to use the theoretical peak performance of the CPUs, since I don't have the machine built and can't test it :). I'm using 2.2GHz CPUs which have a theoretical peak performance of 4.4 GFLOPS per core. So let's do some math and determine the $/GFLOPS

Number of Cores Total
Cost $/GLOPS
(2 nodes) = 4 cores 17.6 $1,491.03 $84.72
(4 nodes) = 8 cores 35.2 $2,505.44 $71.18
(6 nodes) = 12 cores 52.8 $3.539.58 $67.04
(8 nodes) = 16 cores 70.4 $4,563.72 $64.83
Kronos = 8 cores 28.10 $2,490 $88.61

There are several of ways to look at the results. For the same amount of money (about $2,500) we are getting about 35.2 GFLOPS (theorteical peak) instead of 28.1 GFLOPS. Also the $/GLOPS is now $71.18 instead of $88.61 (not a bad improvement in 18 months). For the larger systems, the $/GLOPS is even more exciting. We're fast approaching the $50/GFLOPS mark. Of course, this is still theoretical GFLOPS and more meaningful numbers would be for real HPL performance.

Optional Configurations

In this section I'll look at pricing with optional things such as more memory, faster CPUs, etc. Everyone love speed so I'll look at increasing the clock speed of the CPUs to 2.6 GHz. But there are also times when you need more memory to run larger jobs, so I'll look at those numbers as well. Finally, I'll look at using 16-port GigE switches to go even larger (don't let you significant other see that pile of PCs in the basement).

Faster CPUs

Let's increase the CPU speed to 2.6GHz. The price of the CPU increases to $279.00. Here is the cost breakdown on an 8 node system.

Final Total for 8 Nodes (7 nodes + head node)
Head Node: $932.59
Compute Nodes (7): $4,354.49
Networking: $156.64
Grand Total $5,443.72

This is an increase of $880. The theoretical performance increases to about 5.2 GFLOPS per core (83.2 GLOPS total). So the cost per GFLOPS is $65.87 which is a bit larger than that for 2.4 GHz cores. But still, for a bit over $5,000 you can get 16 cores running at 2.6 GHz!

More Memory Please!

In some cases codes need more memory per node or you may want to run more than one job at a time on the cluster. So, let's look at putting 4GB in each node. I'm using 2x2GB DIMMs that are DDR2-667. I'm also going to stick with 2.4 GHz CPUs to save a little bit of money.

Final Total for 8 Nodes (7 nodes + head node)
Head Node: $1,078.58
Compute Nodes (7): $6,320.67
Networking: $156.64
Grand Total $7,555.89

Notice that the price for this configuration is much greater than 2GB per node. You have 32GB of total memory, but the cost has jumped to about $107.33 per GFLOPS.

Is that a 16-port Switch in Your Pocket?

There may be a few people out there who would like to make slightly larger clusters. But we're limited to a total of 8 nodes because of switch sizes. So I want to look at increasing the total size of the cluster to 16 nodes. For the cost table below I will use the node prices from the first section and then update the Network section.

Final Total for 16 Nodes (15 nodes + head node)
Head Node: $822.59
Compute Nodes (7): $7,681.05
Networking: $371.98
Grand Total $8,875.62

Actually this cost isn't too bad for a total of 32 cores and 32 GB of memory. That's about $277.36 per core. This is a little bit better than the 8 node system. This also works out to $63.04 per GFLOPS which is pretty good (but close to the 8 node configuration).

{mosgoogle right}

What have we learned?

Besides being a fun little "what-if" exercise what have we learned? I will tell what I've learned. First, typing all of these silly HTML tables is a pain. But more importantly, I've learned that in 18 months technology has really changed. For $2,500 we have gone from eight 32-bit cores at 1.7 GHz to eight 64-bit cores at 2.4 GHz, from 2.3GB of total memory to 8 GB of total memory, from one GigE network to two, and we doubled our disk space. That's a pretty nifty increase if you ask me.

So if I were to build a system for $2,500 it would be the 4-node, dual-core AMD Athlon X2 2.2 GHz (4200+) with 2GB of memory per node and two GigE networks. that I discussed above. But then again, I might ask for an increase in budget for the 8 node system (about $5K). The cool thing is that AMD will have quad-core chips out later this year that they claim are drop-in replacements for the current dual-core chips (i.e. no motherboard swap will be needed). Depending upon the price, the upgrade later this year might be worth it. But then again, Intel has quad-core chips out now. Maybe I'll look at an Intel system now....

Jeff Layton is one of the monkeys at He enjoys pining away for ideal home cluster setups that he can build. When he dies he wants to be buried underneath the Playboy Mansion volleyball court (face-up - but his wife threatens to bury him face down if he doesn't behave).


    Login And Newsletter



    Share The Bananas

    Creative Commons License