Suppose you're on the hottest new reality game show on TV. We'll call it, Would Ya Eat It? The host, a man composed entirely of perfect teeth and focus-grouped hairstyle, offers you two choices. You can choose Door 1, behind which is a grilled swordfish steak with a butter sauce. Or you can choose Door 2, behind which is a grilled collection of Everything, found in 10 cubic kilometers of seawater from the North Atlantic. The host points out that Door 2 might contain some lobster, and is likely to contain a much larger portion of swordfish than Door 1. But the catch is that, regardless of your choice, you must eat all the seafood behind that door to win the million-dollar prize. Which door would you choose? I think you'd be a fool to choose Door 2. There's no telling what will be in 10 cubic kilometers of ocean. Depending on where they've sampled, you could get a sea snake, a blobfish, sea star. The dish may be poisonous or toxic. Mixed into that may be some great stuff, but you'll have agreed to eat a lot of things you probably don't want to. Now imagine that this show is about medicine. You can choose between a purified plant product, a chemical, with known pharmacological effects, or you can eat the whole plant to treat your disease. For the most part, I think it would be unwise to eat the whole plant without knowing what else you're getting along with it. And yet, millions of people make the decision to go with Door 2, because they think it's safer to eat a plant than a chemical. How many times have you heard someone say something to the effect of, I prefer to take echinacea because it's all natural, or I worry about how toxic medicines are, so I use B-pollin to treat my asthma? There's something of an anti-medical, anti-science undertone to these kinds of comments. Another favorite is, people have been using spirulina for centuries to control their diabetes. I want to give you three examples of natural products and the evidence about their effectiveness and safety. Along the way, we'll try to separate the wheat from the chaff about herbal cures. The first example is Taxol. And this is really one of the wonderful cases that typify drug development from natural products. It was discovered by the National Cancer Institute at the construction of what is called a compound library from botanical sources. These libraries consist of hundreds of thousands of vials of chemical, obtained in various ways from natural products and represent one of the richest sources of new drug leads. Taxol was a small compound that was a very minor component of the bark of the Pacific U tree. In some early tests of crude mixtures, the Pacific U extract was very effective at killing cancer cells growing in a flask. That got it flagged for further analysis and purification. To make a long and fascinating research story very short and uninteresting, Taxol was finally isolated and characterized after almost 30 years of work. It's been a major innovation and a frontline drug for cancer intervention of a broad spectrum of patients and cancers. If you decided you prefer to get your Taxol from natural sources, you'd need to eat the equivalent of about 60 pounds of Pacific U bark for a single gram of Taxol. Okay, so you're not going to do that, so you decide to make a natural extract of the bark. For that, you'll need ethyl alcohol and a derivative of castor oil, another natural product, called cremaphore EL. This is a very toxic substance and represents the dose limiting component of conventional Taxol therapy, which is just a bark extract after all. Fortunately, researchers have developed two alternatives to the very toxic natural substance. They have a modified delivery system that uses albumin protein found in chicken egg whites to deliver the Taxol, which greatly reduces the toxicity. This formulation is called a Braxane. They've also developed a semi-synthetic form called Taxotir that can be synthesized from the more environmentally friendly English U and actually has a longer mechanism of action and slightly reduced side effects. In the case of Taxol, the natural product certainly kicked off a major discovery in cancer treatment, but the source material, Pacific U bark, is a non-viable therapeutic. Anyone who tries to sell you natural U tree extract isn't doing you any favors. Our next example is St. John's wort. I'm going to use this as a counter example to Taxol. St. John's wort is that rare exception where the herb is sometimes superior to the drug. In Germany, this herbal product is very commonly used to treat major depression. In some, but not all, clinical studies, it was superior to a placebo, had fewer side effects, and performed as well as the pharmaceutical standard of care. We should take this with a grain of salt. Antidepressants are an area of medicine that may not be very evidence-based, so the responses we get from either drug or natural product are going to be strongly affected by the accuracy of the underlying diagnosis. There are some studies that show St. John's wort to be no better than a placebo, including one by the major U.S. altmet group, the National Center for Complementary and Alternative Medicine. It's also been observed that studies in Germany, where St. John's wort is part of the folk medicine tradition, are much more likely to find a positive outcome than non-German studies. This suggests a bias or experimental design issue that should be addressed. The mechanism of St. John's wort is not well known. It's been measured to reduce serotonin reuptake, but the active ingredients have never been successfully purified. It's not without its side effects, although they're usually slightly less than the SSRI-based modern antidepressants. St. John's wort is very toxic to cattle, horses, and other grazing animals. It's especially bad for the young and can cause failure to thrives, skin lesions, and even coma at higher dosages. That kind of information is important when considering whether the herbal therapy is, in fact, safer than the pharmaceutical alternative. The most important human side effect is photosensitization, either directly or through toxic effects in the liver. Exposure to sun can result in skin lesions. The active ingredient in the herbal remedy is found in breast milk, so it should never be used with nursing mothers. You'll notice that these warnings sound an awful lot like the kind of warnings you might hear in a drug commercial. There's a reason for that. Any molecule that inhibits or in any way interacts with biological functions is going to have a side effect. That's inherent in the way our biochemistry is multifunctional. A drug and an herb that interfere in the same pathway are going to have similar side effects. It's only our perception that supposes that because something is leafy and green that it's somehow less toxic.  Those toxins interfere with normal functioning physiology. When we come along and co-op that toxin to provide a specific kind of interference, we shouldn't be surprised that the effects can still be toxic. Red yeast rice is fermented white rice that has a coating of fungus called monascus purpurius. It's used in traditional Chinese cooking and medicines, and imparts a red color and slightly nutty flavor to dishes, including peaking duck and red rice wine. In the 1970s, researchers extracted an active ingredient from the monascus fungus called monocolon K. Chemical tests showed it to be chemically identical to the patented drug lovastatin, marketed by Merck as Mevacor. This was one of the key statin drugs that were shown to lower bad cholesterol and decreased risk of a second heart attack after the first one. The Merck product, Mevacor, was produced from a related fungus mold called aspergillus. Red yeast rice has been very poorly studied in the US and Europe. Clearly, it has a biological effect through the monocolon K, but there are dozens of other monocolons where the effects are not known. They might be beneficial, or they might be very harmful. Unfortunately, the effect of dose required is quite high. You're not going to get in a normal Chinese diet. It must be in the form of supplements. This is why I used the analogy of the game show at the beginning of the video. Our choice is to use the expensive purified product or to take a cheaper, complex mixture of possibly active components. My own choice would be for the well-characterized purified product, but not everyone agrees, and a good case can be made that the cost-effectiveness of red yeast rice makes it a good substitute for those who can't afford pharmaceutical statin regimens. Legally, red yeast rice is in limbo in the US. You can buy it in food form, and there's a gray area of food supplements that don't advertise any benefits. But the FDA has come down on the import of certain products with high monocolon content. There's also a second issue that has to do with adulteration. Due to the cost of fermenting the rice in batches, some manufacturers stretch the product with industrial red dye. Food product regulation in China may not be up to the task of ensuring quality of a product used as a pharmaceutical. Side effects of red yeast rice are similar to the purified statin drug, and can include muscle pain, cramping, and liver damage. To conclude, while natural products are of great benefit to our ability to treat disease, they represent a mix of unknown and known compounds and are not subject to the same level of testing and regulation as pharmaceuticals. That creates a potential risk that might be avoided by isolation and purification of active components. A good case can be made about the accessibility and cost of those natural cures that have undergone rigorous testing, but the potential toxic effects should also be included in the analysis. My own advice, as always, is to ask a trusted physician before you start eating any green and leafy that doesn't come in a salad. Don't be suckered into believing that just because something grew in the dirt, it's good for you. If you don't believe me, go help yourself to a big bottle of castor bean oil or syrup of epicac, and tell me how that works for you. Thanks for watching.


You ready? Hello. Can you hear? Good morning. Thank you for coming. My name is John Dickinson. I am director of technology at Swift Stack and the project technical lead for OpenStack Swift. And with me. My name is Carlos Cabana. I work for IBM on the Cloud Innovation Lab on Object Storage. And one thing that we're going to talk about today is new drive technology and how it works with Swift. Swift turns out to be a really good way to store a large amount of data. And one of the major or the only, the major cost center of a Swift cluster is the hard drives that just by and large dominates any sort of cost associated with Swift. So when you hear new technology coming along, the next thing you want to know is, well, how is it going to work? What do we do? And so one of the new technologies that has come out recently is this thing called shingled magnetic recording. And so today Carlos and I are going to share the results of some of the tests we've been doing at Swift Stack and IBM around this, both kind of at a micro benchmark level and also from an end to end level with Swift itself just to kind of give you some recommendations of what the current state of the art is and what you should do and think about going forward. So the first part of it is what is SMR? What is this whole thing? So shingle magnetic recording. Think about it this way. A long time ago, well, when you think about how hard drives work, you're just like, OK, we've got this spinning platter and you're going to lay down this magnetic stripe on it. And at least in my mind, the way I'd always conceptualize it is that you've got this one layer of magnetic line coming down and then they scoot over a little bit and then they write this other layer and then the next one comes along and the next one comes along. And you're like, OK, just makes a lot of sense. It's kind of like lanes on a highway or something like that. They line up right next to each other. Everything's great. Turns out that's the way they used to do things back in the 1970s. And they do things a little bit differently now. So the traditional hard drives that we're all used to using as spinning media is using something called perpendicular magnetic recording. So imagine those lanes of magnetic flux or whatever it's called laid down on the spinning disk. But actually, what they've done is they've tanked those and laid them on their side so they're vertically. Kind of like do some nice things with the material so the magnetic charge goes vertically instead of just laterally through the surface. What that means is they can get a lot more density because you can start squeezing those together. So we've been using that for like the past 20 or 30 years. Yay. So those are called PMR drives or perpendicular magnetic recording. We're going to refer to PMR drives quite a bit in comparing them with SMR drives. So SMR drives, new stuff. PMR drives, which we've all been used to with traditional hard drives. So what is SMR? What's the difference? Well, what they have done is starting to take those magnetic tracks and starting to overlap them just a little bit to squeeze them even closer together. So in this way, it looks kind of like this. You can think of it very similar to shingles on a roof. So you lay down one line and then you lay down another one on top of it and another one on top of it. And they'll do this for a certain band or zone, an SMR band on the drive surface. And they make a little gap. And they will then start writing down another SMR band. Now, I think that's pretty neat. That's pretty good. They've done some really clever things in the hardware, the actual read and write heads on the spinning media. And you can suddenly write something and then read just a little sliver that's still exposed. Does that make sense? Everybody with me so far? Now, here's the problem. What happens if you want to overwrite one of those things that's in the middle? Well, because the right path is so large, what it means is you're overwriting that stuff that was layered on it. So let's say that you want to overwrite a byte that's right in the middle of one of these SMR zones. You then now have to overwrite. You have to read and then write down again using the SMR pattern all of the data that came after that in that SMR zone. What this means, and the end result here, is that you end up with something that starts looking like a linear access. Because the natural expression of this would be imagine that the entire surface of the platter was written from the very start, very first LBA to the very last LBA on the drive. They would be all completely written with SMR. Then you would, if you wanted to modify one byte, you have to read the entire data, or at least every data, piece of data, after that point on the drive, and then overwrite them all again just to change one byte. So that's why you break them into zones and that's kind of, this is the cost you get of that density. So that being said, obviously this is not going to work just out of the box for anybody writing a new sort of application. So there are basically two different kinds of SMR drives that are available today. There's a third that's not quite yet available. But the two that we're gonna cover today are drive-managed and host-managed. So drive-managed does something very interesting. A drive-managed SMR drive pretends. It just kind of fakes it out and says, guess what? You can use it exactly like you've always been doing. Just plug it in in replacement of a PMR drive and you're not gonna even notice. Well, you might notice, but your applications will still work, you don't have to do anything and the firmware on the hard drive itself manages any sort of translation and if you write to that byte, it's gonna do all of the right things. The host-managed will require the application or the operating system to have those SMRs to know exactly how to lay down the truck. We'll cover that in a little bit. But for now we're gonna cover drive-managed because these are some things that you can actually buy today. To do that, they've got this thing called a media cache. So remember that a hard drive is built out of a stack of platters. The spinning media hard drives are built out of a stack of platters and you can go from the outside edge into the inside edge and you're writing that down. Well, you can see here on this image that you've got these different bands that are the SMR bands. But on this outer diameter, which is the part that's gonna spin the fastest, you've got something that is really just treated as a traditional PMR area and it is the media cache. Now this image I have pulled and there's a link at the bottom from the skylight paper. And there was a paper presented earlier this year at the FAST conference where they took some of these drives, literally cut a hole in it and kind of sealed it off again and then watched what the drive heads were actually doing when they put various workloads on this. And so it's a great source of reference as far as some of the actual limitations of the drives and some of the constraints around it. And in our own testing, we've confirmed a lot of this very experientially. So here's what happens on this and why this is really so important, why I wanted to spend some time on this. SMR drives have something fundamentally new that hard drives didn't used to have. They now have state.  The drive is making this translation between how we used to talk to drives to this SMR way. And what happens is that it now keeps track of what is living where on the hard drive. So you've got basically a map of when I write this piece of data down, now I'd have to find the appropriate SMR track and write it into that and remember where my right pointer is on that and stuff like that. Drives have always had some amount of cache and some amount of RAM on the drive themselves which they can use to buffer writes and cache some reads and kind of smooth out some of the performance. The media cache behaves very, very similarly but it's actually persisted, it's durable and it is written down onto the platter. And it's much, much larger than you would get in a traditional, like the RAM cache. So the RAM cache you might be looking at say 128 megabytes or something like that. The media cache we have tested and found that the media cache will start impacting performance and kind of fill up in enough sense that it will start impacting things at around 30 gigabytes-ish. So you've got about 30 gigabytes on reserved on the overall capacity of the drive as this media cache. So what that means is the media, well the media cache of the drive has to manage two things about the cache. One, they have to manage how many bytes are actually stored in this media cache. As we get too full, then we need to slow things down because the cache, the buffer is filling up so we need to slow client requests in because we can't offload them to the SMR zones fast enough. The other thing is that it has to keep track of say basically the number of writes and so that is something else that can impact performance. So in this case, when you start running out of cache space you're going to have a performance hit and the performance hit starts being experienced around 30 gigabytes of bytes written or about 200,000 writes, whichever one comes first. So if you're writing very, very large files you're probably gonna hit the capacity limit first. If you're writing very small files you're probably gonna run into the number of operations that are being managed. And this is basically the state table that the drive is managing saying that I've accepted all of these writes, these 200,000 writes and now I need to kind of flush them out and figure out how to reorder the writes so that they go out appropriately onto these SMR bands. So that's basically where we are is what the drive media cache does. So that being said, how does this actually affect performance in looking at reads and writes to the drive? The manufacturers will already say that SMR drives are gonna be a little bit slower than traditional PMR drives. This is the results of some tests that we ran at Swiftstock and it is random writes and basically looking at the raw device and then seeking to a random position on that raw device and writing out an object or an object, a file of a chunk of bytes. So direct microbenchmark directly to the drive itself. Nothing else, not even a file system in the way. And so we tested this at everything from 4K writes all the way up to four megabyte writes. And you can see that when we're dealing with very, very small things, we can't really sustain a very large throughput because we end up running into that drive media cache like number of items and then performance goes down. However, when you get to bigger objects, things like four megabytes, then you're gonna be able to start writing much, you have much more throughput before you get impacted by the performance of the drive. And you would think you look at this and like, oh wow, current trends continue. Well, I'll just, man, let's write 100 megabyte objects and we're gonna get infinite speed, right? Not quite. So we suspect that generally the limits that you're gonna see will be at whatever that SMR band size is. So if an SMR band size is gonna be around, say, 256 megabytes or something like that, you're not gonna see any performance after that. And more so, the way I phrase this as a sustainable daily write limit is, in this example, if you have a four megabyte object or four megabyte writes to an SMR drive, you can sustain on average throughout an entire day about eight megabits a second of writes. That doesn't mean that you have to stay at or below eight megabits all the time. You could spike, but within that day, you need to average at eight megabits a second. And that's actually really important because if you do spike above that, then you will start dealing with this background garbage collection, defrag process that's happening to move things from the media cache into the SMR zones on the drive. And when that happens, performance tanks, I mean like three orders of magnitude worse. You can move from writing megabytes to bytes per second. And so that's why you really need to stay below this. Now you look at this and we start thinking about, okay, well what does this mean for Swift itself? So okay, that's not too bad, four megabytes, that's not very big, I've got lots of data, that's a lot more than that. But if you remember back say six months ago in Vancouver when we had quite a few different public service providers who are running OpenStack Swift share a lot more information about their numbers, the object size distribution looks more like this. And this is kind of bad. And the problem is that I'll admit I actually kind of made up some of these numbers to just kind of show a nice big bolt in the curve because we don't have it quite at this resolution. But what we do know for a fact that at least with some service providers is that 91% of their objects are less than 100K. What does that mean if you say that you need to have like four megabyte or more or larger objects? At that point you're like wait, these two things do not work together. So that's going to give you a hint on some of the conclusions that we'll come to in a little bit. But the point is that in general Swift workloads and kind of the public cloud use case and kind of just the general purpose use case, the way a lot of people have been seeing it is that they're very heavily influenced at doing very, very small writes. And SMR drives are very, very good at, we're not very good, but they are much better at doing large writes. And until these things would match up you would say it's going to be kind of tricky to use SMR drives inside of a general purpose Swift cluster. So thinking about that, and we took a cluster and deployed a storage policy in Swift that was only SMR drives. And then we had another one that was only PMR drives. We ran the exact same workloads against both. And this is what we found. So what's important here is not particularly the specific numbers as I want to point out the shape of some of these graphs, especially the one that's in the two wide ones that are in the bottom right. This is exactly what you want to see with a benchmark. You start your benchmark, load spikes up, you've got this nice plateau as you're stressing the system when the system's done, the plateau ends and it's just beautiful shape. So you can see here what I did is I was running a benchmark and I believe it was using some, I think yeah, these were four megabytes. No, these are four kilobyte writes. And I just gradually increased the concurrency. Run test run, then it's over, increase the concurrency, run it again. And you can see exactly what you would expect. Workload goes up, okay great, increase the concurrency, more throughput, increase concurrency, more throughput, everything keeps going. It's like, this is doing pretty well, I like this, I love this. Okay, then I ran the exact same workload against the PMR, I mean the SMR drives. And this is the shape of the graph. So there's a couple, there's two things to notice here. Number one is that first plateau ends up being a lot longer, which you're looking at the 4K writes, it ends up being a lot slower. And number two as I started, as soon as I started to increase the concurrency and do more workload to this going all the way through Swift, it started well and then it tanked. And the performance became very, very erratic, it became much, much slower and frankly just unusable. And so the exact numbers that I got averaged out here is with 4K writes, with concurrency of 200 connections to this particular server that was doing, had just SMR or PMR drives, I was able to get 383 puts per second. It's like, okay, it's reasonable. And with SMR I was only able to get 187. So you're looking at less than half of the performance that was able to do on that 4K writes. So the next thing I did was try to figure out, okay, well what about reads? Interestingly enough, reads were very, very similar. So my exact numbers, I got 1,287 reads per second on these 4K reads. And then I did 4K reads on the SMR drives and I got 1,300 per second. So within any sort of variation according to the particular test, basically they were identical. And then I started thinking, well maybe there's something magic about these 4K things, if that's actually the physical sector size on the drive, well you're not doing anything extra whether it's SMR or PMR, it's just really one operation from the drive anyway. So let's try something different. And I chose some random, not particularly, some like crazy mutual prime with 4K that's around 100 megabytes, 93 or 97 megabyte files or something like that. Increase the concurrency did that and actually saw the same thing. The performance of SMR and PMR with reads was very, very similar, which does kind of show where they might be good for. So all the numbers you saw so far were for the drive-managed SMR drives. However, we also wanted to know what the performance was for the host-managed SMR drives. As you saw in the previous slide, the characteristics of the drives are different. And in the host-managed drive in particular, all the burden of managing the drive and doing optimizations relies on the user or the operating system. One thing John did not mention is that each of those zones that we had in the SMR drive actually have a pointer where you are going to go to make your writes. In the case of the drive-managed drive, is the drive itself who's managing all of that. But for the host-managed drive, it's your responsibility to go ahead and write only starting at that point. If you're going to try to write anywhere else, the drive will reject your request for the write. So what this is doing is the disk is transferring all the burden of all this management to the user level or operating system or file system level itself. The good thing about this, however, is that now the performance becomes much more predictable because you are responsible for it. So ideally, we will test the host-managed drives with Swift itself. We'll just switch the disks and do the same test that John ran. However, that's not possible because now we need to do lots of things ourselves. And Swift is doing much more. Swift is doing operations on reads and writes, of course, on directories on symbolic links and metadata on the files. If we were to go ahead and implement all of that for the host-managed SMR drive, we would be getting very close to some sort of user-level file system. However, we really care about the performance for the reads and writes. So for that purpose, we wrote a simulator that will drive writes and reads and read writes in a specific ratios to try to understand what are the performance characteristics of this drive. So as you can see here, we have a simulator section that we can configure to do reads and writes in any ratio we want. They send all the requests to a processing module. This module, what it does is for the reads themselves, they will call directly to the drive libraries to make the read for the writes. It's going to use some internal buffer that we configure between two and four zones in size. And when those buffers are complete, there's one part of the process that is actually doing the write to the drive. They are for optimizing the writing on the drive as much as we can. At the same time, you will see on the right side that's the fragmentation process. So due to the sequential nature of the writing of the zones, one thing that may happen over time is that you will not only be writing files, right? At some point you're going to delete some of them. And that will leave you with fragmented zones. And you don't want that. What you really want is one to have things as compact as possible. So this fragmentation process, what it's doing is reading from the beginning of the drive, compacting all the data, removing all those holes that you're having there, and writing them at the end of the drive. And it's running at the same time as we are doing all these other operations. As you see here, all of these operations are going through a set of libraries. Those are called ZBC libraries, which come from the manufacturer of the host-managed SMR drives. So we're not calling the drive directly, we're calling the drive through these libraries. So these are the absolute numbers that we get for writes, for reads, and for write-heavy read writes on the host-managed SMR drives. Our intention here is to try to see what is the best that we can get from these drives, but not under the best possible situation. So we are dealing here with very small objects which are somewhat detrimental to all the reads, because it's forcing the drive to go and read very small chunks of data all the time. Let's start with the writes, which are at the top. The numbers that we get are fairly stable and very close to the rated write speed of the drive. Now, when we begin to disturb these writes a little bit with the reads, of course, we will see the performance degrade a little bit, because after writing a certain amount of data, we're going to introduce a few random reads there which are going to take the head all over the disk, and then we're going to continue to the writes. The intention of that is to see is somewhat more normal usage of the drive and under these scenarios. And at the bottom, do you see the reads? And the reads are performing fairly flat, and this will become important in the following slides. But one thing that we learned from the manufacturer here that is very important at this point is that the libraries that handle the drives, they don't do any caching or any prefetching at all. So for every read operation that we are doing, we are actually issuing a SCSI command making the disk respond to that, bringing it back all the way up to the application and then continue. The disk is not trying, sorry, the libraries are not trying to be smart and read more than one track and more information as the operating system would. Therefore, the consequences of that is that with this testing scenario, the reads do not scale. But it's not only about the host-managed SMR drives. We also want to compare with the PMR drives because we have something running today. We're going to know how does it compare with this new technology. So we did a read-rides and read-rides comparison between these two types of drives. What we can see here in this chart is the performance results for the reads. So the red line that you can see, the one at the top, is the performance with the PMR drives in what we call no-direct access. So I need to point out, what we're testing here is the same simulator that we had before, but running on the SMR drive and on the PMR drives, simulating SMR drives. And there are two ways in which we can do that. We can make the access to the PMR drive as any application while trying to use all the optimizations of the operating system that means caching and prefetching as well. Or we can do a direct access to the disk, which is much more closely to what the libraries in the SMR drive are doing, right? So we wanted to do this comparison. We see the performance with the PMR drive, what we call no-direct access, which is using all the prefetching, using all the caching, scaling very well up to roughly 45 simultaneous processes. Then we see the middle line, that's the SMR drive, that scales fairly close to the PMR drive up to about 10 processes, and it flattened out from there. And the reason for that is what I mentioned before, that these libraries are not doing any caching or prefetching at all. So it's reasonable for them to scale up to a point and then flatten out because you can really perform any better than that. At the bottom, you will see what we call the direct access to the PMR drive. And what this is doing is you're sampling we're purposely disabling all the caching to all the caching being done by the operating system when accessing the drive. You can see this is very flat here. And the reason for that, and we learned this recently, actually, is that this is being purposely protected by the mutics on all the operations and it's being done at the level of the libraries themselves. So what's happening is this is not going to scale at all. So unfortunately, that's showing a somewhat artificial way in which they are protecting all these operations. So it's not a fair comparison with a direct access to the drive, unfortunately. So these are performance numbers for the right. What we can see for both the PMR drive and the SMR drive is that both are performing the right very close to the rated specifications of the drives. Clearly, we'll see that when we're doing direct access to the PMR drive, it will lower its performance, right? But all of them stay within certain parameters. And the one thing that we like to point out is how flat the SMR drive is. It's fairly constant. And as I said before, it is very close to the rated performance for the drive. And finally, we want to do the read-write comparison. And notice here that the moment we begin to interrupt the writes on the disk with some read operations, we can see both the PMR and the SMR drive performing fairly close. So some conclusions that we need to get from all the slides that I just showed you. First, we're dealing with the first version of these drives and the libraries themselves. They did not do any caching so far because this was the easiest and the simplest implementation that they thought they could have right now. They're leaving all that burden for the moment to the user, though I'm pretty sure there are lots of opportunities for improvements in those libraries with respect to caching. And the other thing that we see is that it's very clear compromise that we want to, do you may want to have here, right? We're talking about what performance versus what density do you want to have, right? It's either one or the other clearly. And there's potential for having very good data density in these drives as well. However, as we can see here under a right heavy read-write scenario, we can see that this performance fairly, fairly close. So that leaves us with the big question. What do you do? You're a deployer of Swift and you've got drive veneers saying we've got this new hotness. It's this greatest thing.  So in general, based on our testing, you're gonna, I think, not be surprised by some of the stuff we'll say here. So our recommendations are that number one, in a general use case scenario for Swift, you should not use SMR drives. The performance hits that you get on those, especially when you're dealing with the smaller objects, especially as we've seen the distribution of object sizes inside of Swift clusters, is not worth the additional capacity that you can get by the increased density of those drives. But drive manufacturers actually aren't even selling these as general purpose drives, so they're not lying to you. It is very, very good. These things are sold as archive drives. These are things that are designed for a particular use case. So there are some use cases that this could be very good for you. And specifically, it would be do you have a workload that generates large files and large being four megabytes or larger files? And it does something that writes and potentially can read. Reads, remember, can be okay. For example, maybe you have backups. I know there are a lot of different backup vendors out there and I've worked with quite a few of them. But specifically, I know that Commvault has the ability to have very large writes. And with backups, you're looking at kind of this worm data. You write it once and you read it many times. You don't try to overwrite. You don't try to keep updating the data after you've written it once. So it ends up being a pretty good use case for SMR drives. Another one would be potentially like video surveillance cameras and things like that. They generally can output about one megabyte a second. And so that would be a great use case for SMR drives, especially when you configure those to say every minute or every 100 megabytes or so, cut it off and write a file down, persist that and then keep streaming to the next one. And other use cases that I've seen to kind of like the big data, I think genomics and DNA things and you've got these huge data sets. I think the genomics thing is very interesting because you've got these very massive data sets. I would not put the SMR drives in the path of the high speed analytics and comparisons and they're kind of like very small files that are reading and writing things. But that full sequence genome that is gigabytes in size, not a bad case for SMR. So the point is there are actually some reasonable use cases that people are doing today that would be an interesting fit and something to consider for SMR drives. And if you wanted to start doing that, if you wanted to start experimenting with this and figuring out, okay, well, I might want to gradually ingest some of these SMR drives into my cluster, how would I do that? Step one would be set up a storage policy inside of SWIFT. We've got the storage policies that allow you to isolate a particular set of drives from the rest of them. So you would set up a storage policy that would be set up as this is gonna be my backups storage policy or something like that. And then as you continually, as you generally buy or as you gradually buy the SMR drives, you can put those into that storage policy alone. Then any kind of data that's written to that storage policy, you just make sure that the applications that are writing into that storage policies are the ones that are doing the large files, not a lot of overwrites doing the big files. Yeah, it's kind of that sort of thing. So like I said, I think that there are some good use cases out there that you could, but probably not for a general purpose. Great, let's just start buying SMR drives because they're gonna have a little bit more capacity available today. But that does mean that what does the future work? What can we do to actually improve this and make this a better story for everyone? And so there's quite a few things that I think that are obvious directions. Some of them aren't very simple, but some of them are fairly obvious. One of them is something that I think that we're very interested in Swift overall anyway. And in fact, some of the Rackspace developers have been working on their Go language hummingbird server have already implemented some of this. And that is being able to restrict down the number of connections to a particular drive. In general, if you said if we could rate limit a particular drive such that every day it's only gonna average eight megabytes a second, then look, we're right in the zone of what's good for SMR drives. Now, if we could bring that in and make that a little more configurable to the operator and to the deployer, I think that's something that would be widely usable for a lot of people inside of Swift. So all the work we did so far, so we were using the drive management SMR drives so certain performance and we did a lot of work ourselves in order to manage the drives themselves. However, the next logical step which is being developed nowadays are file systems that are aware of the SMR drives. So they will go and do all these optimizations themselves. So what we need to wait for now is for these file systems, at least the first versions to be available and get them and run all these tests again and see how those file systems perform because many people will use them for sure and they're going to really solve some of the problems that we saw during our tests here. And then I think finally kind of the pie in the sky multi-year timeframe horizon of what could be possible was is that after you have particularly done some small level rate limiting to a particular drive after SMR file systems have been developed and you've tested those and deployed those and those have shown promise, well then what would possibly be next? And at that point you're getting into the realm of things like well, how would we teach Swift itself to speak SMR? And really what you're talking about here is almost substituting SMR for any sort of native memory, native media conversation between Swift and whatever that storage media is. And frankly, it's terrifying, it's a huge, that would be a huge multi-year effort inside of Swift that I don't think anybody is chomping at the bit to try to tackle, but because a lot of that will be taken care of, as Carlos said, by those SMR file systems to start with. So it's something that's definitely a possibility as far as this far future thing and if you really want to commit some developers, several years of time to work on this in the community, patches are welcome. But that being said, I think that to summarize, what we've got is some interesting ideas for the future work that some things that we, the developer community, can work on. We've got some good recommendations for what you today can take advantage of in that, namely, don't use SMR for a general purpose, but use them for a specific thing. Use a storage policy inside of Swift to say that here are large files that have not really much overwrite, not a huge, ingest bandwidth requirements. And those are things where you may be able to see some opportunity of benefit for SMR drives. And that being said, I think we've got a couple of minutes left. If you have some questions, we've got a couple of mics. There's one in the back, okay. He's bringing it to you, Jerry. An idea of the improved density that you're seeing with SMR over PMR? Today, SMR drives are about a 20% improvement over PMR drives. For example, you can go by an eight terabyte PMR drive, you can buy a 10 terabyte SMR drive. And those numbers are gonna pretty much increase along those lines, but I've been told that it's gonna be about a 20% improvements over intensity. Does any of this change when you go to like a kinetic interface instead of a SCSI interface on the drive? In other words, SMR behind kinetic, have you thought about that? So SMR behind kinetic drives, I don't think, to be honest, I don't know. Nobody's tested that. Nobody's built kinetic drives that are SMR as well. I think we should do one new thing at a time. But that being said, I would anticipate that it would be basically the SMR, kinetic would be the host aware thing talking, it just happens to be on the exact same physical thing to the SMR drive. So I would imagine that you would see similar performance as the host aware drives. But that's just because there's already an API translation layer. But I don't think that those even exist right now. So I'm just speculating on that point. Anyone else? What do you think about cash tiering from SMR to PMR? Cash tiering, as far as saying that I've got my colder data that I'm gonna go put on SMR and then my more active stuff will be on PMR. And then my super active stuff will be on, Intel's 3D cross point, of course. You know, all the new hotness on storage media. Yeah, I think that's fine. I think that's outside of the realm of the actual drive. But that's something that is actively being talked about what every week probably within the community, people come in and say, well, I've got this data and I wanna move it around. And this week here at the design summit, one of the things that we are preserving a large block of time to talk about is exactly, okay, so data movement inside of a cluster. And tiering is one expression of that, of saying I need to move something from this place to this place, how can we do that as efficiently as possible? So I think that they're related, but not particularly dependent on one another. And if there is a good cash tiering or data tiering solution with a Swift deployment, whether that's inside of Swift or external to Swift, then I think that SMR drives would definitely have a place inside of that. I mean, it's kind of a generic answer, but I think it's something to consider, yes. Maybe we're time for one more question. Hi, just a quick comment. I'm Rick Wheeler from Red Hat and I've been working on SMR for a couple of years with the drive vendors in the Linux community. So there is a lot of work going on in these special file systems, but more specifically, probably the most interesting one is Western Digital has a device mapper module that they haven't quite pushed out to open source yet that will let you do effectively host-managed SMR on normal file systems. So that was talked about at LinuxCon in Seattle and what we talked about at Vault, so something to watch for, yeah. I think that's great to hear. And one thing we didn't really spend any time talking about were these host-aware drives. Yeah, because they're not quite available yet, but there's kind of like a mid-tier in there. So I'm really excited to see that. And in fact, I'm very grateful for being able to talk directly to those drive manufacturers, I know Carlos is too, as far as we've received a lot of insights. We had a link to the Skylight paper, which is, you can see a lot of great numbers. I've got a link on this at the bottom here, which is these slides that are available if you wanted to study that in more depth. And I know that at Swiftstack, and I'm sure also at IBM, there are quite a few people behind the scenes who were involved in a lot of this testing. I want to say thank you to them as well. Thank you very much for coming. Thank you. Thank you.


Greetings and welcome to the Introduction to Astronomy. In this video, we are going to discuss the expanding universe, and we'll look at how we discovered this and what it actually means. So, let's go ahead and get started. And first of all, how do we discover the expansion of the universe? Well, this actually goes back into the early 1900s, and one of the people studying this was Vesto Sliffer. And what he was doing was to study actually looking for planetary systems. So, he was collecting spectra of the spiral nebulae. Now, he was working at Lowell Observatory, which is known, which is the area where Pluto was discovered in 1930. And he was looking for planetary systems, that something that the Lowell Observatory was concentrating on, looking for planets. And he was obtaining the spectra of these, and what he found is that most of them exhibited large red shifts. So, what he'd see is this is the pattern you should see for the spectral lines, and the patterns were all shifted. Everything was shifted to the red. And, as you may recall from the Doppler effect, this means that the galaxies are moving away from us. So, he found that all of these galaxies were moving away from us. And, George's Lamatra in 1927 published a paper suggesting an expanding universe, which was something, interestingly enough, consistent with general relativity, although Einstein had added in a cosmological constant to make it so that the universe did not necessarily expand, but it was something that general relativity could have predicted. And he used Sliffer's observations as his evidence. So, this was some of the very earliest evidence of the expansion of the universe. Now, also sometimes known for talking about the expansion of the universe was Edwin Hubble. And Hubble, in 1931, published a paper relating velocity of recession, so how fast a galaxy was receding, to the distance of the galaxy. And, what he published, his original data from 1929 is shown here, and that's only this very small portion for very nearby galaxies, only going out to a few million light-years. In 1931, he had added more data, and now going out to a hundred light-years, and was able to see a very distinct relationship between the velocity here on the y-axis and the distance on the x-axis. Now, this was very important because it becomes another way to determine the distance. The velocity is very easy to measure, and once you measure the velocity, you can run across and down and find the distance using this graph. So, the better we can determine this line, this becomes a new method of determining distances in astronomy and can work out to the very edge of the universe. And he gave us what is now known as Hubble's law, and Hubble's law simply says that the velocity is equal to a constant times the distance, meaning that if you wanted to find the distance, you just take that distance is equal to the velocity that you measure divided by Hubble's constant. And that was, again, would be a way to determine distances to the galaxies. So, first thing we have to do is understand a little bit about what Hubble's constant means. And what it is, Hubble's constant is actually the slope of the line that we saw in the previous slide. If we can figure that out, we can then use this to determine the distance to any galaxy that we can get a spectrum of, and that is beyond our local region of space. Our current estimate of Hubble's constant is about 22 kilometers per second per million light-years. Very strange units, not something that we're used to. What does this mean? This means that a galaxy at a million light-years away would be receding at a speed of 22 kilometers per second. A galaxy at 10 million light-years away, 10 times further away, would be 10 times this. 10 times 22 kilometers per second, or 220 kilometers per second for its recessional velocity. This is, again, just the current estimate, and it is still subject to revision, but over the last few decades we have been able to narrow down this value more and more with more detailed measurements. How about some variations? How constant is Hubble's constant? Does Hubble's constant vary? And one of the questions can be, is Hubble's constant really a constant? And what we have to remember is that when we look out in space, we look back in time. So what we mean now is that if we are using this to determine distances, which is what we want to do, it means that we have to make the assumption that Hubble's constant that we're using was exactly the same 10 billion years ago as it is today. If Hubble's constant has changed over that time, then this equation will not work. So what kind of things might cause this change? Well, gravity is one example. We remember that galaxies are pulling on each other, which should slow down the expansion rate of the universe. If gravity were the only force involved. So that means that galaxies should have been moving faster in the past and slower now. So that means that Hubble's constant would have changed just because of gravity. Now there's a lot more going into this, but just to keep that in mind that Hubble's constant may not really be a constant. So what do we mean by the expanding universe? Let's take a look at this here. And what we mean by a uniformly expanding universe, first of all, to start out, what this means is that what it means for expansion is we come back to what we call the Copernican principle. And we give Copernicus credit for this because he was the one who moved the Earth away from the center of the universe. And what the Copernican principle means is that we are not at any special point in the universe. And when we look at this, it can be a concern when we look at all these galaxies receding from us. Does that mean we are at the center? And that's actually not true. Every observer would see exactly the same expansion. So it does not matter which galaxy you are on. The galaxy here sees the other three galaxies in this image receding away from it. But if you were on this other galaxy, instead, you would see the same thing. You would see all the other galaxies receding from you. So what it means is the galaxy, the universe, has no center. So there is no center to the universe. Everyone will see exactly the same expansion. Let's look at a couple of examples to try to understand this a little bit better. We can look at an example of the expansion here. Let's look at a one-dimensional example, first of all. So a one-dimensional example we can think of as a ruler. And that's just because it's very simple. We look at an ant on a ruler. And if we look at the 2 centimeter mark, so our ant right here, we'll see all the other ants moving away and the more distant ants moving away faster. So this one, the ant here at the 4 mark, there's our ant at 2 and there's the ant at 4. This one expands at 2 centimeters per minute. So now it's gotten larger. So while this ruler still looks the same, now the centimeters are actually twice as large. So he has moved away at a distance of 2 centimeters per minute. The ant here at the 7 centimeter mark has now expanded at 5 centimeters per minute and the ant at the 12 centimeter mark has gone at 10 centimeters per minute. So, but it does not matter who you look at. If you pick out this ant, this ant still sees everyone else expanding away from him. So it does not matter where you happen to be located. Each of the ants is going to see the exact same thing, is going to see every other ant moving away from them and is going to see the most distant ants moving away faster. Now that's a one dimensional example. Just looking at one, let's go ahead and look at another example which would be a three dimensional example. And this could be what we call raisin bread. So baking some bread with raisins in it. And if you imagine as the bread rises, the raisins will then spread apart. And what we mean is that an observer on any of these raisins would see every other raisin moving further away and the more distant ones moving away faster. So this one here, we look here at the central raisin that we pick out and that's now over here as it expands. This one that as it expanded, this one moved from three centimeters to six centimeters. So it moved three centimeters away. This one that was way over here at 20 centimeters away is now 40 centimeters away. So while the universe expanded, things changed by greater distances. What was close to you is only a little bit further away. What was further away is even further away from you. So the expansion is the same regardless. And once again, it does not matter which raisin you pick. We picked one here, we could have picked another just as easily. But the view would be exactly the same and that's what happens to us. Within our galaxy, we see every other galaxy moving away from us. But if we imagine an astronomer in a distant galaxy looking at all of the other galaxies, they will see exactly the same thing. They will see our galaxy moving away from them and all of the other galaxies. So no matter where the observer is, they're going to see exactly the same thing. So what does this mean for the expansion? If we know this, what does it mean? What is really expanding is the space. The galaxies are not getting bigger. Galaxies do not get any larger. The size of the galaxy stays exactly the same as seen in the image here. The size of the galaxy is still the same. Galaxies are gravitationally bound together, so they do not take part of the expansion. What really expands is the space between the galaxy clusters. It's not the galaxies, not the clusters themselves, nothing that is gravitationally bound together. For example, within our solar system, the planets are not getting further apart by expansion. They are all bound to the sun by gravity. But when we look at the very largest clusters of galaxies and superclusters of galaxies, those are getting further and further apart. Now, one of the problems here is that in reality, there is no edge or center to the universe. So we do not have an edge, we do not have a center. In our examples, that doesn't really work out. If we look at our examples, we did have an edge to the raisin bread. And you could imagine that someone at the very edge, at the edge of the loaf, would see things a little bit differently because you'd be looking out into emptiness or into something else. Now, in terms of our universe, that's not how it works. So our universe would not have an edge or a center, but would actually involve an additional dimension. So you can imagine that the universe, we have a three-dimensional universe expanding into a fourth dimension. And that means that we're never going to see any kind of edge or any kind of center to it. Not in any dimension that we as three-dimensional creatures can actually observe. So let's finish up here. Space itself is expanding. It is the space that is expanding and is carrying these galaxies and galaxy clusters apart as it expands. So let's finish up here as we do with our summary. And what we've looked at in this section, first of all, the expansion of the universe is something that we've seen for the last hundred years. So we're able to get the very earliest detections of that. Edwin Hubble gave us a formalized version of his Hubble's law that related the velocity of an object to its distance. So giving us another way to be able to determine distances and one that worked out to the edge of the universe. And then to emphasize the expansion is an expansion of space, not a direct motion of galaxies. It is only the space between the galaxy clusters that is actually expanding. So that concludes this lecture on the expanding universe. We'll be back again next time for another topic in astronomy. So until then, have a great day, everyone, and I will see you in class.