A fiery horse with a speed of light, a cloud of dust, and a hearty hyosilver, the Lone Ranger. With his faithful Indian companion, Toto, the daring and resourceful masked rider of the plains led the fight for law and order in the early western United States. Nowhere in the pages of history can one find a greater champion of justice. Return with us now to those thrilling days of yesteryear. From out of the past come the thundering hoofbeats of the great horse, Silver. The Lone Ranger rides again. Come on, Silver. Let's go, big fella. Are you still there? Then read the Lone Ranger's 14-year-old nephew and his horse, Victor, were in a large cave. They were there to await the return of the masked man and Toto, who had gone to a nearby town on special business. It was dusk outside, but in the cave it was quite dark. Dan had been lying on a blanket dozing when he was roused by the clatter of hoofs on the rocky floor of the canyon outside the cave. Calle, Victor. Maybe that's our friends. No, no, that doesn't sound like Scout and Silver. Someone else is coming this way. You and I better get to the rear of the cave. Can't take chances on being seen. We might have to answer too many questions. Come on, Victor.  Well, here we are, Pedro. This here's the cave I was telling you about. See, my friends and your kids. Are we following? Nope. I reckon we got away clean. By net I'm glad to see that they're cave. I'd sooner stay there for now on and go back to that there circle bar. Only time, Pedro. I try to tell you it's no good to work. We get the job. What happens? We shoot that. We're most killed. Better it is to stay away from work. Yeah, that's just fine. The trouble is I got the habit of eating. See. Which same calls for cash. Well, this, my Pedro, needs your horse into the cave. Leave the way, my good friend. I'll follow. You don't think of that. Come on, dear. That, that, Pedro, it was all your fault. Huh? We had good jobs at the circle bar. You went coyoteing around. You blame Pedro for the trouble. That's wrong. You're with it. I hope we weren't followed into the cave. Oh, no, amigo. I don't think anybody can follow us. Well, here we are. This is a loon. It's a big cave. Also dark. Yeah, I'll strike a match. There is water to be found? Plenty. There's a spring here in the cave. The Lone Ranger showed it to me. The Lone Ranger did, huh? Yep.  He sure was good to me. I'd give him all I got in the world. Just a minute. What? You're nice. You can't see me. I'm back in the cave. But I can see both of you. Who are you? Now, if we don't mean no harm, we're just law-abiding citizens who've got more trouble than a bob-tailed horse in fly time. You just spoke of the Lone Ranger. Well, he's my friend. He showed me this here cave. Good. Here's a candle. Well, it's only the point. I'm nearly 15. My name's Dan Reed. Well, you can call me Pete. This here's my pal, Pedro.  How do you do? By Ned, now I get it. I remember hearing about a lad by the name of Dan Reed who travels once in a while with the Lone Ranger. And I remember him telling about Pete and Pedro. Well, my boy, we had bad luck. We had jobs. You two were working? Yeah. What sort of thing is he here about? Say, Dan, where is your mass friend? Where can we see him? We sure need his help right now. You know there's trouble piling up at the circle bar. He'll be back here, but maybe not before tomorrow. Oh, doggone. What happened at the circle bar? Well, Pedro here, he and me got new jobs and figured we'd like to look around the circle bar spread. That is, Pedro was the one who wanted to snoop around. Oh, you're all the time. Anyhow, the ranch is owned by a widow named Pettengill. She's living in a small house. It's sort of new. And right nearby is the old ranch house that's not being used. With his cyclone cellar. Cyclone cellar? Yep. Under the old house. Well, we figured, I mean to say Pedro here, figured to look that cellar over. We went there and done so. I wish we hadn't. Why? Well, a kind of what we've seen there and what happened next. What did you see? Rifles. Case after case full of army rifles bearing the government marking. Then the shooting began. Yep. Someone opened up. So we lit out him fast. Wait. Oh, God. What's the matter? There's a United States Marshal in Red Oak who's trying to learn about some army rifles. Someone around here's been buying them. Oh, my. The lone ranger in town went into town to meet him. They're going to help him. I just no wonder we were sure that. We thought too much. You've got to go back and keep an eye on those rifles. I'm sure not me. But now that you've seen the rifles, they'll probably be moved to another hiding place. I'd be sure if we went back there. Well, then. Then I'll go. Now, hold on, lad. I'm not known at the Circle Bar. I can get a job there. Don't do it, Daniel. There's some mighty tough armies on that spread. I won't get into trouble.  Tell him what you told me and tell him where I've gone. He'll have my hide for letting you go. No, you won't. He'll be mighty grateful for what you've discovered. Well, I sure owe him a plenty. You know, many is the time he got me out of trouble and Pedro can say the same. Yes, sir. We're both mighty deep in debt to that marriage man. Dan reached the Circle Bar soon after darkness and found little difficulty in securing a job. It was later the same evening when the lone ranger and tunnel made their plans with the United States Marshal, a grim-faced man named Daniels. The three were in a clump of cottonwoods at the edge of town. Tonneau, don't do this unless you understand that there'll be great risk. Me savvy. Both Tonneau and I will go a long way to keep the stolen rifles out of the hands of Indians. That's right. Oh, by the way, Daniels, what makes you think the rifles are somewhere in this vicinity? Well, it's a long story of tireless investigation. We've had a lot of men at work on the case. I see. We finally got confessions from two guards who looked the other way when the rifles were stolen. They could tell very little, however. They didn't know who paid them. Oh? They were paid in paper money. It came through the mail. And it was mailed from here, from Red Oak. You said you thought the rifles were still under cover. Yes. At least none have been seen in the hands of Indians. And we assumed that they were stolen to sell to the Indians. Well, Daniels, my instructions from Washington said to help you in any possible way. You approve my plan to try to smoke out someone? It might work. All right. Now, Tunnel, you know your part. That's right. And you understand the danger. Here's the gold. Put it in your pocket and head for the cafe in Red Oak. Let everybody there know you have a lot of cash and that you're waiting to meet someone. You'll keep an eye on things, mister. Yes, Daniels. And I guess we're ready. I'll see the sheriff and he can make the arrest. In the Red Oak cafe, the bartender was busy polishing glassware and serving his customers. His face was without expression until he saw the stately figure of the banker. Oh, Mr. Gregson. Great day. I don't often see you in here. What do you have? Uh, clear sort of one or... Oh, doggone it. There's an Indian in the place. I hope none of the boys make the mistake of serving them drinks. Especially right now, we're going to have this Marshall town. Yes, I talked to him. Who, the Indian? No, no, the Marshall. Looking for a shipment of rifles that was stolen from the Army. No, is that so? Yes. There's a soda water. Now, Indian, I can't give you... Me got gold here. Well, even so, I can't serve you nothing. Me want paper money. You want folding money? Uh-huh. You mean you want to change the gold for paper? Not right. Here's a sack. You wait. Is that sack full of gold? Isn't that right? Very interesting. Just a minute, Baldi. Let me see that sack. Now, sure, if I wasn't selling the Indian anything... Never mind that, Baldi. You, Indian, where'd you get all that gold? Well, me come here to meet man. Make trade. Yeah? What kind of business? Who'd you come here to meet? I mean, I'll tell that. Now, don't get proddy, Redskine. I can take you to jail and hold you for questioning as a servant. Me not talking. You not make me talk. All right, then. The possession of that gold is reason for holding you on suspicion. Wouldn't you say so, banker? Yes, yes, sir. I quite agree with you. Come on, Redskine. And you give back gold. You not take gold. Stop struggling and do as I say. If you're on the level, you got nothing to worry about. Now, come on. Come on. The lone ranger and Daniels were outside the cafe, not far from the swinging doors. It was too dark for anyone farther away than the marshal to see the lone ranger's mask beneath the brim of his hat. Someone's coming out of the cafe. Yes. Tuddle the sheriff. Oh, oh, oh. Steady boy. You happen to know either of those men who are dismounting at the bar. I'm sure you know. I know. I know. I know. I know. I know. I know. I know. I don't know either of those men who are dismounting at the hitch rail. No, it healed. No, I just wondered if you did. I don't want to stay in town long. We've got a busy day tomorrow. Yeah, we got that new tender foot to break in. There were just what our whistleman had to head back with a circle bar. That new boy's young, but he looks strong. Yeah, I didn't get his name. Did you? Read, then read. Daniels. I got to leave you. There's one thing that comes ahead of everything else. Got to get the silver ride. But say it. Oh, my love. Oh, no. Racing away from town, the lone ranger headed for the canyon. In record time, he drew rain at the mouth of the cave where he had left Dan Reid. Dismounting, he ran into the cave. Then stopped short when he saw his old friends, Pete and Pedro. Pete, Pedro. You got here fast. We're sure glad to see you. Amigo. Hey, senor. Where's Dan Reid? Well, he read to the circle bar. He told us to wait here for you. Why? Well, we found some rifles in the cellar of the old house. And then someone shot at us and we lit out. We took a look at it. And then we found some rifles in the cellar of the old house. And then someone shot at us and we lit out. We come here to hide and found young Dan. Rifles, you say? Say, senor. Well, Dan knew you was interested so he went there to keep an eye on him. Hey, where you going? We got more to tell you. Later. Here's your big fella. One cellar there. While he rode, the lone ranger made hurried plans. He pulled off his mask and tucked it beneath his shirt. Then he stopped and dismounted long enough to take materials from a saddle bag and darken his complexion. He looked like a different man when he once more leaped to the saddle and rode on toward the circle bar. As he knew the house, he heard an alarm being sounded. A woman was shouting and men were rushing toward the smaller of two houses. Come here to the house. Something terrible has happened. The lone ranger dismounted and moved toward the house with the ranch hands. Miss Madam, Miss Pattengill was happy. Durit, bring all the boys inside the house. Get them all. We're all here, Miss Pattengill. All but Slim and Lefty. They went to Red Oak for the evening. Where's that new boy hired this evening? Here I am, Mrs. Pattengill. Well, come in here. Come into the house right away. Durit, all the cowhands aren't here. I told you about Lefty and Slim. Then there was those two collutes that lit out Pete and Pedro. They lit out? When did they do that? Well, it was before dark, ma'am. Before I lighted the bunkhouse lamps. Didn't even take their gears. Then there are the crooks. Durit, I've been robbed. What? Robbed of all the money I'd saved up to pay off my mortgage to Mr. Gregson. I had it all together and those two crooks stole it.  Miss Pattengill, I've tried to warn you about hiring men. I'm your foreman. Why don't you let me do the hiring? Me, I suppose I should. But Durit, when someone needs a job, I can't turn him away. We'll find those two crooks. But wait, those two crooks! What's that, boy? Oh, um...  We're going to spread out and find those two. Meet me at the corral for fine lord as we go gunning for those crooks. All right, Durit. You read. Wait a minute. Yes? You're not going with them in. You go to the bunkhouse and wait for me. I want to ask you a few questions. You better have the right answers. The curtain falls on the first act of our Lone Ranger story. Before the next exciting scenes, please permit us to pause for just a few moments. Now to continue our story. Durit, the circle bar foreman, went to the corral with the ranch hands to instruct them in the search for Pete and Pedro, accused of stealing Mrs. Pattengill's money. Dan Reed, as ordered, went to the bunkhouse to await Durit and answer questions. Answer a few questions. What do you mean by that? Dan. You. Recognize the clothes, but your face. I was but a word of disguise when I came here to look for you. Did Pete and Pedro tell you I was here? Yes. As soon as I arrived, I heard about the robbery. I was at the door of the house while you were inside with the ranch hands. I know Pete and Pedro didn't steal that money. They ran away because someone fired at them when they discovered army rifles. Yes, Dan. I thought those might be the rifles you were looking for. I thought I'd better come out here and try to keep an eye on them to see they weren't moved. All right, Dan. Someone on this ranch knows a lot about those rifles. Whoever it is, he doesn't want Pete and Pedro to tell what they know. You know what I think. What, sir? I think Mrs. Pettengirl's money was stolen so there'd be a search for our friends. They'd probably be shot on sight. What are you writing, sir? Dan, we've got to stop that manhunt. The foreman is meeting his men at the corral. I'm going to try to hold their attention to give you a chance to get Victor out of the corral as quietly as possible. Then what? Don't take time to saddle your horse. Bear back. Ride to Red Oak in a hurry and give this note to the sheriff. He'll tell you why Tonneau's in jail. Tonneau in jail? Yes. And if you meet the United States Marshal named Daniels, trust him. All right, here's your note. Come now, Dan. I'll take Silver and join that group. There's so many, man. I don't think I'll be noticed. I'll go around to the far side of the corral and get Victor. All right, Silver.  You see. The boys of Mrs. Pettengirl don't get the cash back. She'll lose the ranch and we'll all be out of a job. You know what? Those two galutes are the ones that took it. They wouldn't hide until they'd have been in it. All right, George. I told you how to split up. We'll reach pairs of traffic. There's a lot of questions to trade on. So be on the lookout for signs of any place that might be camping. Now then. Hey, just a minute. Who are you? I've been to Red Oak. What about it? Just this. There might be a connection between something that happened there and the robbery that took place here. Who are you? What's the difference? Who's the top hand around here? I am.  You got something to say. Say it and be quick. We got work to do. What I have to say needn't be hurried, dirrick. You don't have to send the men out. How's that? You'll understand when I'm finished. Well, then get started. An Indian came into Red Oak a little while ago. He had quite a bit of gold. He did. He wouldn't tell the sheriff where he got it. You suppose he stole it for Mrs. Patton Gill? Well, I don't know how much did he have. I don't think it was way to counted. Well, he's doing it. Hey, you kid, where you going? Let him go, dirrick. I told him to clear out. You did. Say, who are you anyhow? Well, he went through that once before. Hey, you, I told him. Derrick. He's got away. Let him go. What is the town with me and bring the sheriff and Indian back here? I'm not leaving this ranch unprotected. Unprotected? Are these men here? I don't know who I can trust. None of them been here long. And if I send them into town, they can tell the sheriff what's happened. That's an idea. You heard that, boys? Go tell the sheriff the whole story and bring him the prisoner back here. Now, get going. Right around him. Right around him. Get around him. Get around him. Get around him. Get around him. Get around him. Go, you're the only one that's been here for any length of time. What about it? That was pretty flimsy, Derrick. What do you mean? You know, of course, that the money the Indian had wasn't stolen for Mrs. Patton Gill. Just what are you getting at, mister? Maybe if you'd left the ranch, one of the boys would have been curious about the cyclone seller in the old house. The cyclone seller? You wouldn't want to take a chance on having someone see it, would you? I'll never go near the old house. I've never been in the cellar. We'll take a look at it now. See here?  We'll... we'll look at it, Derrick. Oh. So that's what it is, huh? Yeah. That's right. I'll take your guns and no one gets hurt. Now, look. Mrs. Patton Gill gave strict orders about the old house. She moved out when her husband died and it hasn't been entered since. Maybe the seller can be reached from outside the house. We look around. All right. That's the way it's got to be. The old house is right in the back of the new one. I'll show you where. There is a way into the old seller from the outside. I didn't realize I don't see how I shot to be fired in the old seller without being heard by Mrs. Patton Gill. What's that? Don't you know? I sure wish I knew what you were driving at. Let's wait until we're in the seller. He's the outside door. Double door is wide enough for a wagon. Open one. All right. It's dark in there. Here's a candle and matches. Light up. All right. There. Ground slants down. A wagon could be running here. There is a wagon over there. Go ahead and see what's inside. Damn. Good for footprints on Derrick. Yeah, that's right. That'll prove I never was in here before. Look in that wagon. Rifles. Cases of them. Are you surprised? I sure am. Hey, this isn't... You hit? No. The candle was knocked out. This way. Hit some bottles or something. The door is closed. Look, you can't see out there. Keep low. I'll return that fire. No. Kill that marksman and we won't find out who's running the rifle deal. What of it? You get out if we don't get him. Be quiet a minute.  There'll be others here in a few minutes. All right. We'll wait. Hug the ground, Derrick. You can't get us in total darkness. For what seemed an endless length of time, the tension in total darkness continued. At the slightest sound of movement, the heavy gun would bark in the cyclone cellar. But every shot went wild because of a simple trick by the Lone Ranger. While he spoke clearly telling Derrick to hug the ground, his hand guided the foreman to the top of the cases in the wagon. The shots were low, far too low to find the mark, but the killer was determined and kept firing. It became a battle of wits. The Lone Ranger might have shot at the flames from the unknown enemy's gun, but instead he bided time. He kept close to Derrick and in the darkness trusted that he would not be made the victim of a chance bullet. In the meantime, Dan Reed had delivered the message in Red Oak, then hurried to the cave where Pete and Pedro waited. The Marshal sent me here. He sent for me to take both of you to the ranch. Ah, no, Shiree, not by a jugfish. But the Lone Ranger's at the ranch. Well, that makes it different.  We've got to travel. The spasmodic gunfire in the cyclone cellar continued until the sound of horses' hooves came through the night and then it stopped. The group of horsemen with Marshal Daniels in charge came to a halt and dismounted at Mrs. Hepton Gill's small house. You're going to wrap on the duo, Marshal? Yes. And I'll do the talking. The rest of your boys keep still. Is this here Redskins' ghillie? Shiree put him in my custody. Oh, thank goodness, thank goodness, you boys are... who's this? The name is Daniels, ma'am. United States Marshal. Do come in. There's been the most awful gunfight going on. Where gunfight? It was muffled, but I knew it was guns. Over on the cellar of the old house, I don't know what it means. I do. I've been so frightened. I went to the bunkhouse, but none of the men were there. Please look in the cellar of the old house and see what's going on. We can tell you what happened. You bet we can. Of course, what happened? That man's mask. Mask? Don't mercy me. Speak up, Derek. You're the foreman. Put his mask on in the cellar and I don't blame him for suspecting me. Of what? Boys, there's a wagon load of brand new army rifles in the cellar. Of course, ma'am. Don't let him in. Huh? Yes, Marshal. Mask man thought I was working with someone to sell him to the Redskins. And I can see why he thought so. I was the only one that had been on this ranch any length of time. Then when I wouldn't leave here tonight, looked more never like me. It was nearly a costly mistake. Then he took me down there. The shooting started. We were trapped. If he hadn't signaled me to climb onto the wagon at the same time he said to hug the ground, we'd have been killed by the spray of bullets on the floor. I guess the killer thought we were dead. Where is this killer? Very damn. Look at our shoes. Yeah, muddy. Now look at Mrs. Pettengill's shoes. You, Mrs. Pettengill. All right. You're smart. Don't any of you move. I'm leaving here on the handiest horse and I don't aim to be caught. You won't get far, Mrs. Pettengill. I'll risk that. One thing I wanted about you're stolen money. I suppose you faked that so you could pretend you were going to lose the ranch and discharge all the men before your customer came to get the rifles.  It also gave you an excuse to hunt down two harmless men who had seen the rifles. I'm not admitting the thing. It doesn't matter, Mrs. Pettengill. You see, banker Gregson is already in custody. What? He stepped right into a little trap when he called on this Indian in jail. He thought the money the Indian had was to be used to buy rifles. After questioning he broke down completely and confessed that he was offering contraband rifles for sale. Why is that? He told us you were the one who engineered the robbery of the rifles. That's a downright lie. Gregson is the one who engineered it. He's the one who can name a couple of no-account crooks that did the stealing and brought them here to my place. Thanks for the information. Maybe you've got Gregson, but you haven't got me. I'm leaving here and I'll drop the first man to come through this door. Hey, Pedro, look at Mrs. Pettengill holding it out. I'll take that gun. No, no. Let me go. Let me go. Now you'll be firing on any of us. You fool, Pete. Why did you have to come just now? Well, Pedro, it looks like we can never do the right thing. This time? What? She here, mister. This is Mrs. Pettengill you're holding the gun on. That's right. She's under arrest. Yeah. I'll admit that, mister. Maybe you've got me for this job, but maybe I won't be in too long when I tell how Gregson managed everything. Held a mortgage to dangle over my head to help me make up my mind. Well, it'll be for the law to decide. Hey, boys, where did that masked man go? He and the Indians stepped out a minute ago. Hey, there they are over yonder. Dan is with them. Hey, there, Dan. We're traveling, Pete. We'll meet again, Daniels. Come on, Phil, there. Get him up. Oh. That masked man. Marshall, I'd sure like to know who that masked man is. I can't tell you his name, but he's called the Lone Ranger. This is a feature of the Lone Ranger Incorporated, created and produced by George W. Trendle and directed by Charles D. Livingston. Tonight's story was written by Fran Stryker. The part of the Lone Ranger is played by Brace Beamer.


OK, everybody, settle down. Probably we'll start now with our next speaker. And by the way, there is like charging sockets in the desks. If your laptops died, you can charge them probably. So yeah, now let's move on and meet our next presenter, Brin Reeves, who is from the LVM and device mapper team, who'll tell us more about DM stuff and some good hints. So the floor is yours. Thanks. Let's welcome him. Hi. Thank you for the welcome. And it feels like a real honor to be talking to you all in such a great venue today. I spoke last year remotely on work we've been doing on device mapper statistics. Hopefully in person is even better than via remote stream. So we shipped the first version of DM stats a couple of years ago, 2015. Quite a bit of work since then. Before I jump into things, can I get some quick shows of hands? Who uses device mapper, either multipath, LVM, whatever? Yes, good numbers. Good. Anybody played with DM stats before? Anyone taking a look? Yeah, yeah, got some users. So what I wanted to do today is look at the most recent work. So apologies for those of you who haven't seen the earlier stuff. I'll do a real quick recap, but there's a lot of interesting new stuff. And that's really what I want to focus on. If anything seems confusing, because you didn't see last year's or you haven't had a look, track me down outside, catch me on a mailing list, IRC, whatever. I'm always happy to talk about this. It's kind of my baby at the moment. So the recap then. DM stats is a flexible IO statistics framework for device mapper. It builds on kernel work done by Mikolash Potoka. It's a really great mechanism that we have in the kernel. I'm heavily indebted here. It gives us a great base on which to build really powerful, interesting user space work.  We then turn those into metrics. So we do rate conversion, unit conversion, other things that you don't want to do in the kernel, as well as kind of metadata fields to help you understand what the information means. Some of the advantages that device mapper statistics have over the conventional PROC disk stats, what you get from SAR or the IO stat program, we can optionally do nanosecond precision counters rather than the Jiffy millisecond precision that you get traditionally. We also can do these funky little latency histograms. Now, I did promise him my abstract to demo them. Then I remembered I demoed them last year. So again, due to shortness of time, I'm not going to cover those again, but again, do feel free to catch me. I'm happy to show them off. The really cool thing, I think the really cool thing, we don't just have a single set of statistics for your whole device. We can dice and slice your disk in a whole load of different ways to produce much more fine-grained results. This gives us an insight into what's going on inside the device that just never was possible before. There are a couple of things in the talk I'm going to claim as world firsts. I should qualify that. They were possible with things like detrace, system tower, also by parsing block trace data. But these are easy to use, prepackaged solutions that you can deploy in production with no fear of excess resource consumption. The kernel acts against excessive memory consumption in the stat subsystem. So these are pretty solid compared to some of those earlier solutions. We also have really flexible reporting thanks to a colleague of mine, Peter, in the team and the device mapper reporting engine. You can choose whatever fields you like in any order, mix, map, recombine, anything. It's really cool. The downside, well, it's not such a downside. You do have to configure this. So let me just qualify that with a real brief demo. By default, we don't collect anything. So I just configured DM stats. It really is as easy as that. Now, this is taking a whole lot of defaults. If you add more options, set other flags on the command line, you get more control over the process. But this will set you up for a basic set of reports on all of your device mapper devices. So I'll just kill those all off now. We'll come back to the demos as we hit specific topics later on. So if you've ever unpacked an LVM2 or device mapper table or checked out the Git tree, you'll have seen the what's new and what's new DM files. So this is a kind of we snapshot of the stats changes from the last year and a half, I guess, but really most of it landed last summer through to now. And there are a couple of bits that are still to hit master. I'll give you all the branches later on if you want to take a look at those. The plan is those are going to hit master and find their way into a release. Should be in the next few weeks, maybe a month or so tops. But they are all there if you want to check them out. Some of the other components I'll talk about later are currently being developed out of tree. The idea is to merge them once they're stable. But if there's demand and interest, I'm quite happy to make coppers available for those so that you can get easily installable versions of them too. So the additions compared to what I discussed last year, we added a grouping facility. So you've got these regions on your device. We can now glue those together and get aggregate stats for them. I'll explain the regions and areas concept again in a moment, just again to recap. It is the distinguishing feature of DM stats or one of them. And it's an important concept to understand to really make the best use of it. So my baby, the thing I'm quite pleased with is file mapping. We'll take a deep look at that in a moment. But the basic idea is you can have IOS stats now for an arbitrary file anywhere in the file system, as long as it's EXT4 or XFS. So if you have virtual machine images, large database files, you needn't confine your workload in order to measure the behavior on just those files. You can just zero in directly on the data that you're interested in. I think this is a really powerful facility. It's very new. So obviously, I want to hear your feedback. I want users. So do please, if it sounds interesting, take a look and get in touch with any comment or feedback that you have. I'll explain more about the monitoring demon that comes with that and why it's necessary later. There's also a little, started off really as a toy to solve one particular customer problem. But I think it's going to be generally useful. The original name of it, well, my boss thought it looked like a rude word. So I came up with the name IOScope. It's a way of visualizing your IO. So you have a disk. We want to know where is the IO going? Where are the hotspots? This allows you to do things like cache dimensioning to figure out where bottlenecks are occurring in your workload. Perhaps you have a directory that's severely overloaded and there's contention to perform updates on the disk there. There's also some general API improvements. The metric interface had a bit of a revamp. So you can just call a single function now with an enum value and get your data rather than this kind of plethora of different name functions. The old interfaces is still there, of course. We don't just rip things out because we have something better unless there's a real need to. With all of these new things, we have a lot more output now, especially with the file mapping with files that have lots of fragments, extents in them. And this kind of adds some pressure to your terminal, especially if you're on a big screen like this. So we've added a filtering mechanism which works with the aggregation subsystem. So you can pick the level that you look at your data. Do you want to see the very smallest bits, the areas? Zoom out one level and look at the regions or take a more global view and look at whatever groups you may have configured. So, quick recap. I should have switched those slides around, never mind. Quick recap on this region and area concept. A region is just an arbitrary range of a device. One or more sectors anywhere in the device and we'll collect statistics for that according to the area configuration. In many uses, a region will have one area. So there's no further subdivision. But for some uses, you may want to chop the region up into yet more fine-grained units. Each one has its own set of counters within the kernel and you can report that data individually at that level of granularity or glue it all back together and look at the overview. This gives us a very granular range of possibilities. We can go all the way down to a 512-byte block or all the way up to a multi-terabyte disk. If you don't configure device map of statistics, no regions, no costs. Actually, Nicholas can give you the true story on that. I believe it's just a conditional branch. Yeah, so if you don't have any regions configured, we just look at the pointer. If it's not set, there's very, very minimal overhead. I guess you could compile it out of your kernel if you wanted to, but I would tell you I'm missing out if you do that. So, let me just pop back to that one. So what comes in the box? Actually, I haven't finished putting all of it in the box yet, as I said, but this is what the box will be in a couple of months' time. So the first thing that you'll find is the DMStats command line tool that I was using a moment ago. That's the primary interface to the subsystem, at least for kind of administrative use. You can build applications on top of that. Currently, the CAPI is fully supported. That's ready to go for production. There is also a Python API, which is... I told somebody the other day it was 80% complete, and then I thought again, I think it's more like 70%. But the plan is to get that finished off this year. DMStats I've been working on for two, three years now, and this year should be really the last year of heavy development on the call. We're getting very close to feature completeness now. The Python API, I think, adds some really interesting possibilities. I'll talk more about those later on. One of which is the ioscope. Right now, it doesn't use the Python API. It was actually the reason I wrote the Python API. Again, more about that later. Those last two are currently on GitHub. The URLs will be up later. I grabbed them from the PDF when it's published. I would try and write them down now. So do feel free to come and join in on GitHub. I'm ever eager for users. We do respond to issues and pull requests. So if these are interesting projects, then come and get involved. Okay, we've done that. So this grouping concept, this kind of sprang up out of the initial idea of mapping objects in the file system and being able to look at those as a single object, even though the data is scattered out across the disk. You can use it manually. It's quite a lot to configure. I was gonna do a demo of this, but it is manual. It's lots and lots of steps, and it would be kind of boring. But what I will do is take that demo and put it in the GitHub Wiki. So if anybody wants to take a look at those steps, there are also examples in the manual page of using the group facility manually. It's real simple though. You wanna make a group, DM stats group. Your regions list is in the same format as kernel bitmap list. If you've ever set a CPU mask in the kernel, like one, two, dash, four, I borrowed the code from them, so it's exactly the same. This identifies the set of regions we're gonna group, and we can give them a friendly name and alias. Quacks here, call them what you wish. When we do file mapping by default, it's the file name. It's just a helpful label to appear in reports to allow you to identify what it is that you're looking at. So, ungroup, you just give the group ID, which we've returned from the group command, and it dissolves the group back to its component regions. Why do we want to use grouping? Well, I kinda hinted at some of it already, but let's just put it out clearly. DeviceMapper and LVM device segments were one of the first cases for this, so I'm sure the number of hands that went up for using DeviceMapper, you'll have seen a logical volume that perhaps has either been extended over time or for some other reason contains segments that correspond to different areas of a physical volume. We can map those and group them together so that you can either look at the logical volume as a whole or drill down to hey, why am I getting bad performance on this area of the disk and see those separate from the remaining segments? That then kicked off this idea that, well, hey, these aren't the only structured objects that you find inside a block device. We can do other things, files, extended attributes, directories even. You may want to look at the journal, perhaps your journal is a bottleneck. Right now, the only one of these that's implemented in mainline is, as I said, the file mapping. The idea for the others is to sort of defer them to the Python interface so that we can have a generic way. If you have some object of interest and you know how to find it, you can write a little bit of Python script and we'll be able to find your objects and report information on them for you. Again, more on the Python bits later. Database formats, device data and metadata. Another colleague of mine, Milan, works on DMCrypt and I was looking at some of his work the other day and thought, hey, we can do a stats layer for this.  Another use case is virtualization. The rev and overt, I think overt still does it this way. I haven't checked recently.  Sometimes you'd like to just say, okay, that one VM image which is spread out here, just give me the information for that. And with DMStats grouping, all of this becomes pretty easy. So we'll take a look at the first of those use cases next, the file mapping. As I said, it is kind of one of the features I'm most pleased with from this round of work. Again, hopefully easy to use. The create command allows you to create all sorts of regions and other configurations. We had a new switch here, dash dash file map and this is gonna go ahead and create those images for you, create those configurations for you. It does accept globs as well as single files. So if you've got a whole directory full of VM images, just start on image or start up QK2, whatever it is and we'll create new mappings for all of those. By default, we stick them together in a group and we set the alias for the file. If you don't want that for any reason, perhaps you want to do some customization first. Just set no group and we'll report all of the unique identifiers for them. Bear in mind with big files, you're gonna have a lot of identifiers to work with there. It's probably more suitable for scripting than manual use. So real quick, look at that then. So I've got a little Fedora 25. Am I gonna be able to get into Vert Manager here? I should be able to. Yeah, there we go. So I'm just gonna create a file mapped image. As you can see, my file system has quite a level of fragmentation. It's not a huge file and there are 927 extents in it. That may be because I have intensely tortured this file system because the more fragmented the better test case it is for my work. But this is a pretty extreme case for such a small image. So now I've got that there. Let's, and this is gonna be a little bit minced and mangled. I apologize for that. The reports are quite column hungry. They just love horizontal space. So if it comes out a bit, don't worry too much about it. Again, I'll post some nice clean output or you can take a look at the man page which has sample output. It's not too bad. Let's just boot that machine up if we can. So we should see that. That was the boot loader being read out just then. Hold on, let me filter this some more. Okay, so now we're just seeing the group, as I said, using those filtering options. The machine is still booting. You can see there's quite a lot of reads going on there. Not much in the way. Oh, there was a little right there, 1.8 bags. Hopefully that demonstrates the ease of use for this facility. I'll stop that now, because it is kind of hard to read and just delete that mapping. I use that command a lot. It just resets everything to the ground state and lets you go ahead and create a new configuration. Okay. So as much as I say it's my great new thing, it's for a user, kind of as simple as that. Again, quick show of hands. Does this look interesting, useful to people? Do you think you may have it used for this? Brilliant, give me some feedback. I'd love to know what else you would like to be able to do with it. We have thought, could you repeat the question? Oh, me repeat the question, I am not very bright. Sorry, the question was, could we integrate this with DM cache so that you can say cache this particular file or some other hint to request the data in the cache? Actually that specific thing, not so much, but one of the things we are looking at is to use DM stats for dimensioning your cache. But you've now given me an idea and I'm gonna go back and think about that some more. I'll need to talk to Joe, who is the architect and author of DM cache. I can't even give you a guess for the feasibility of that right now. I'm not familiar enough with the internals of the cache, but it's certainly an interesting question, thanks. Okay, yeah. I'm just about to come onto that, no problem. So when we run that create command, we take a snapshot of the extent. Now that's not a snapshot in the sort of thin piece ends, just a momentary consistent view of the file. But of course, files change. And a QCOW very often starts off very sparse. There are some limitations here. I'm gonna draw those out clearly in a moment. I don't wanna sell this as a universal solution. It works very well for many use cases, but there are some things you need to be aware of. We, if we are right into the file and it's not fully pre-allocated, we're gonna need to update the mapping. Now there's a command in the tree now. It's a new DM stats command update file map, which just does a one shot update. The next bit of work to land again, hopefully in the next few weeks, adds a demon, a DM file map D, which just sits in the background. It's a tiny, tiny demon. It has like one file descriptor and all it does is read the extents and then talk to device map to update the mapping in memory. This allows us to follow the files as they change and grow, adding and removing extents as needed. This facility has two ways of operation in the demon, two modes of operation. We can either follow the inode or we can follow the path. And these are useful for different styles of updating the files. This is why you kinda have to choose here. If your file is likely to be moved or renamed and you want to continue to get updates for it, you want follow inode. And we will keep that file descriptor open on the very same inode number and follow it as it moves around the file system. On the other hand, if your update style is to write a new file and then unlink and replace, you want to use the path following mode. And that will just constantly reopen the same path and update the statistics mappings accordingly. So we'll have a quick look at how to use this and some of the command line options in a second. I just wanna run through the limitations first and discuss a thing that I call the right gap. It's important to be aware of these, but again, as I say, there are a lot of situations where these problems are not going to affect your statistics or the use of the facility. We do require physical FIE map data. FIE map is an IOCTL that gives us the location of a file's data on desk. Most file systems that implement it tell you where it is on a physical block device. XFS, EXT4. I list those two here, but actually as long as you're not ButraFS and you have that data available, DM stats file maps will work. ButraFS gives us these virtual internal devices because of its internal volume management features. And we can't tell where the data is. So it's unlikely that this is going to be supported certainly anywhere in the short term for ButraFS. We also can't map the space until it gets allocated, right? This, you know, until something causes those, those disk blocks to be allocated, we can't see them. So as we said, sparse files, anything that you're extending via right past EOF or if you're using the FAllocate whole punching and extension API, then you will want to use the demon or the update file map command to make sure that your, your data, your mapping is consistent with, with reality. It's also important to be aware, and this is where the right gap comes from, that the updates are asynchronous with respect to that allocation, yeah? So a right happens, we'll take a look at this in graphical form. It's easier to understand in a second and sometime later we react to that and it will always be this way. There is no way I am hooking into the block layer and holding the world up while we update our mapping. It's, you know, these are statistics. They're not mission critical. They're not something to, to harm the performance of the system to, to collect unless you're doing very kind of specialist research or something. So the right gap is my term for this, this interval between a right triggering an allocation and us responding to that. We use the Inotify API. I did actually write it to use FA Notify first because it's new and cool and turns out FA Notify doesn't really deal well with file deletion whereas Inotify does. That's the reason in case you were wondering. So when a right occurs, it triggers the allocation. It will complete at some point. And at that point we get the notification, excuse me, that something changed. I notify and FA notify our VFS interfaces so they don't really tell us that allocation happened. So we have to do some heuristics, stat the file and things to figure out what's going on. And if something changed, we'll call this API routine DM stats update regions from FD and that carries out the FIE map and then the update steps to re-synchronize things. Now I've used slightly facetious units on the diagram of Femto Fortnites. It's kind of hard at the moment. There is such a gamut of storage performance from at one end NVMe Express which is down in the sub microsecond completion times all the way out to crappy consumer flash with sucky flash translation layers where it just goes off to lunch. Sucky FTLs we cope really well with but the shorter your device response time, either the more lag there will be between your updates or the more resources in terms of CPU you have to throw at the daemon to keep everything kind of synchronized for you. Why that 20.23 centimeters? I've deleted that thing a dozen times now. LibreOffice. Okay, so that fundamentally is the right gap. I have some kernel patches to address one aspect of it. I'm not gonna talk too much about that because it is kind of bleeding edge and again we are short of time to fit everything into the talk but again if you're interested just get in touch. Wrong way. Let's just have a look at the daemon before we head on to ioscopes. So I mentioned there's a daemon. Actually it was running, I hope. Yeah, there we go. So here's dmpharmapd. It is a daemon so the command line is kind of cryptic. It's explained in the manual page and there are examples of how to run that. I think rather than me trying to type all of that in now I'm just gonna do a blue peter here and here's one I prepared earlier. Let's have a look. Okay, so this is a little bit of debug logging. Is that reasonably readable? I'm gonna kind of pick out highlights anyway but actually we've got some, there we go. That should be a bit better. So what this is saying, dmpharmapd is monitoring farmer script of three. You can see I'm using a shell redirect. This is how you would run it from the command line. It starts up, does a couple of quick checks and it finds out that it's monitoring vm.image and if I just scroll down through here we should see the file getting extended. So we've, that's where I want to be, we've returned from the monitoring function and we've got something to check. So we scan the extents, we get the disk locations printed out there. We kept region zero, which was one out of two of the extents and we found three new ones. So the file was growing pretty quickly. Actually I kind of paused the demon to make sure there was some good churn for you to see there. Again, most users are never gonna be interested in seeing that level of detail but just to explain it, if you ever need to debug it or collect data to send back for feedback. So that's file mapping, as I say, it's really new. I think it should be useful. If you run into any problems or so on then do get in touch. We are actively looking to improve. I think we have about 10 minutes left so I'll do a quick demo of the IOScope and then like a really brief mention of Dumpy, the device mapper Python bindings. I'm really excited about that project but I started it after I put my proposal in for the talk so it's kind of not fair to hijack the rest of the talk just because I've got some cool new thing I've moved on to. The IOScope, as I said, it's a visualization tool primarily. It does little ASCII graphs. It also generates data in CSV and JSON time series format. That enables funky 3D plots. Unfortunately, there's still a bit bleeding edge. Rendering them with Matplotlib live is kind of slow. I hope next year if I'm invited back to speak again we'll have a better solution for that that I can show off to you then. The idea with this is we can easily identify hotspots. You can literally look at it and say, hey, there's a big peak there and lots of stuff is slamming that region. So far as I know this is a first as a general purpose facility in an everyday OS. I remember some guys at EMC in Ireland showing me a DMX frame doing this about 10 years ago in court and I was like, whoa. So I'm really, really happy to now see this coming into general use for DeviceMapper. It is a hacky Python script at the moment. I make no apology for that. Half of the script is this kind of rough Python abstraction over the DMStats command and DMSetup. And that was what prompted me to write proper bindings for the whole DeviceMapper library, including DMStats. So once I finish on that, I'll port or rewrite DMioscope to use the new interface. And once it's stable and we're happy with it we'll merge that back into the rest of the DeviceMapper project. So before I run out of time completely, let's, which one do I want to be on for that? That's this one, isn't it? Okay. So first of all, I'll demo the static mode. This is good for analysis purposes if you're gonna crunch the numbers later because it's unchanging. We have a fixed number of bins, just mentioned bins, and I didn't mention histograms yet. Fundamentally, the Ioscope is a histogram. We use the region facility to chop your device up and we accumulate counts in the bins and then we plot them out graphically. That's all it is. Since the bins can be variable size, technically the values are frequencies, not counts, but for the most part you needn't worry about that unless you are processing them numerically. So I'll start that running and let's get some load. So you can see here that the display is updating in real time behind me. Then the vertical rows are the disk space from zero up to 28, 30 gigs, whatever my disk is. And along the horizontal axes are the frequencies or counts of data. Since these are fixed sized, there's no difference between those. The frequency in the count is equal. On the other hand, if I stick the adaptive mode on and this is a bit, yeah, there's a whole lot of options involved in this and I want to reduce those down and make it easier to use. But without getting too detailed, we start off with just one bin and we use the counts to adapt. So we merge things if they're a low count and we split them apart if they're a high count. And you can see that it's kind of scrolled off the screen because I've got it zoomed in so far. This is handy for, we really have no idea and you just want to get a rough idea of what your distribution is. You can just leave it run. There is also a summary mode which rather than giving you an instantaneous point in time snapshot, it accumulates everything. And that's handy to just get a rough sense of what your workload does when you're really starting from nothing. Okay, so I've got, I think, five minutes left which just gives me just a moment to talk about my new thing which is feel free to call it dmpy if you like. I had to call it that because there's already a pydm which is another shell wrapper for device mapper but this is native. It uses libdevmapper, it's written in C, supports Python 2 and Python 3 from a single build. We're aiming to get 100% coverage of at least everything that makes sense from the API and I want it to be a comfortable discoverable Pythonic kind of interface. It maintains as little state internally as possible. Everything it gets live from the underlying library, there is a cache in there but it's a reference cache so that we avoid recycling Python objects when we don't need to. That's all the caching that we do. There's also a little state machine in there. There has to be a state machine because the C API has rules and Python, if you break the rules, you want an exception.  So the state machine is essential to prevent invalid uses of the C API. It's, I think, useful for prototyping, for one-offs like the IOScope which has kind of evolved past a one-off now but all for interactive use. You can fire it up in the Python shell and just shoot commands into there which, come back to that in a second. Python import. Dump it as dm, help dm. And this is one of my things with, if I'm gonna do Python stuff, I'm gonna do it Python style. Doc strings, discoverable, introspectable. You will find that these are very, very similar to the text that you'll find in the libdevmapper header file, the comments that document that interface, but obviously with names and so on I change to suit the Python use. I won't spend too much time on that. I'm sure you can read docs as well as I can. Okay, so. The classes and modules that it currently provides, the basic familiar, dm task, info, cookie, timestamp and so on, cookies really broken. I didn't understand dm cookies until I wrote this code. Amazingly, they still work, but there's some weird clutter in the interface that I'm gonna clean up as soon as I get home. So hopefully you won't see that. The stuff I really was trying to focus on, dm stats came in later obviously once the basic infrastructure was there, but it maps directly to its C equivalents. And I'm about to run out of time, so my last thing to mention is the dumpy test suite. This lets you do unit testing for device mapper in Python. So again, we're not gonna have time to run through this in detail, but here is a bit of Python script to do device creation with Udev synchronization. It's a lot easier than doing it from C, but you have the same level of control and power you'd have with the native API. Okay, I had better draw things to a close there. That's just a bit more information about the test suite behind me there. So just about fitting into, oh, future work. Yeah, shrink the right gap, dm stats, numpy integration, probably. I talked about that, right, I did. So if I don't call it to a close there, we're gonna have no time for questions at all. So thank you all for listening, thank you for coming along today, and I'll turn it over to the floor for questions. No, but you could feed... Sorry, the question was, you can use globs for file maps. Can you also do recursive? I guess you'd use like a bash process substitution with a find for now. I don't think I really wanna add file system recursion to that code. The shell does the glob for us. That's coming soon. I'm gonna build a directory mapper on top of the Python work, but files came first. I think your hand was next. The question there was, what's the overhead? It's not each trivial to measure. There are two sources of overhead. There's overhead in the IO path, which Miklash will be able to tell you in more detail and more correctly than I can without kind of going and looking. It's relatively minimal, but it scales in the number of regions. Unless you're using histograms, or it scales with the bins, but I have tortured the histograms and I can barely measure them with Perth. So it's pretty low. On the other hand, if you create a million regions, that's gonna have some expense associated with it. There's some optimization we can do that's on my to do in terms of how we communicate with the kernel when there are very large numbers of regions. But the more you throw at it, the more the cost. But the unit cost for single sets of regions is really quite hard to even measure. I believe it is linear, yeah. And it won't allow you to use more than, I think, 25% of physical memory or 50% of Vmalloc space. Yeah, you should call in your file system maintainer and blame them. Sorry, the question was, if the hotspot identifier finds a hotspot in file system metadata, what should you do? And I've just totally passed the buck on that. Honestly, it depends on the file system, it depends on the device it's on. It's a big topic to get into here, I think. But it's a valid question and it's something we may include in the Wiki or the documentation later. Sure. Yeah. Okay, so the question was, if I find a hotspot, can I find out what file it is? Yes, you can. I don't have anything kind of canned for that, but yes. If I don't file an issue on GitHub first, just to remind me, it's pretty easy to do now. It's just a case of asking the file system, basically, and the file systems can all tell us. Yeah, yeah, that's a good RFE. I didn't say that it wasn't expensive, but the computer does all that for me. Yeah, yeah, yeah. Yeah, the forward look up, we need to reverse that and it will be costly, but. Oh, you can do it already, cool. I probably need to talk to you in that case then. Yeah, I think we can use that. Any more questions? Yeah, yes, it works with all, sorry, the question was, does it work with thin volumes? We support all of the device mapper targets. Originally, there was a limitation for request-based device mappers, a multi-path, but the kernel side of that was solved a couple of years ago. So any device mapper target, linear, multi-path, snapshot, old snapshot, new snapshot, thin provisioning, cache the lot. Yeah, thanks. Any more? Yeah, yes, absolutely. Yeah, I hope again, if I'm invited back next year, once we finish kind of the nuts and bolts and mechanics, we can start to talk more seriously about, I'd really like to do a workshop next time, take a kind of two, three hour chunk to look at the different ways you can actually use it. You know, I'm talking about the code here rather than the performance investigation for possibilities that it really opens up. Yeah, thanks. Right, yeah. So each region has a unique identifier within that device. They're not unique across different devices, but if I've got my root volume, region ID zero there is a unique identifier for that region. We also, in the user space, create these fake identifiers for the areas. It's just an index, you know, but it's the region IDs and the group IDs that you use to tell the command line tool or the library, hey, this is the group or this is the region that I want to operate on. And they are qualified by the device. Sorry, I'm tired. The question there was, what are these identifier things that we use to reference the regions or the groups that we're operating on? So the question was on integration points suggesting the cockpit dashboard system and also to integrate with virtual machine start and stop. And those are both great ideas and it's the thing I'm most interested to see now is to kind of get this plugged into different places. The simple text displays I've been showing you can be turned obviously into really fancy graphics and I'd love to see that. Yeah, the question was on containers using Oval AFS. I don't know is the answer to that. If there's a device map or device underneath, we can work with that. I have no idea how the file mapping stuff would work with Oval AFS. Red Hat's Docker stuff has tended to use the device map driver for that. So I have limited personal experience of Oval AFS. I don't wanna make a promise, I don't understand yet, yeah. Anymore? Right, well thank you for coming and putting up with me and my continual forgetfulness to repeat the question. Thanks again. Yeah, sorry, he was a teacher before, he's strict.  No, no, it's fine, I really hate people who overrun and I'm gonna nip out for five minutes because I really need nicotine, but I wanna see yours. Ah, come on. Hey. It's only been 15 years since I'll be in one stance. Yes, yes, I finally get to close, I may even have closed it already, but yeah, it's one of the oldest bugs.


General Mills, makers of Cheerios, the Oats cereal ready to eat, and Whitties, breakfast of champions, presents, The Lone Ranger! A fiery horse with a speed of light, a cloud of dust, and a hearty hyose silver! The Lone Ranger! With his faithful Indian companion, Toto, the daring and resourceful mask rider of the planes led the fight for law and order in the early western United States. Nowhere in the pages of history can one find a greater champion of justice. Return with us now to those thrilling days of yesteryear. From out of the past come the thundering hoofbeats of the great horse Silver! The Lone Ranger rides again! When Bill's that fat, the kids all shout, you can't strike that slugger out! He gets a hip because he knows, he's got go power from Cheerios! Yes, he's got go power! That's Cheerios, the cereal shaped like little letter Oats, and those Oats stand for oats. The good grain Cheerios is made from. Every delicious spoonful of Cheerios and milk is real muscle building food. Each spoonful contains vitamins, minerals and proteins your body needs. Yes, those good things in a Cheerios breakfast do good things for your body. Help you have healthy nerves, good red blood, strong bones and muscles. You can see that Cheerios is made to give you real go power. So make sure you have a Cheerios breakfast every day. Then you'll hear people say... He's feeling his Cheerios! The Lone Ranger and Tonto rode the trail outside of the town of Dove Creek. Suddenly... Into the woods, Tonto, quickly! Come on, Tonto! Come on, Tonto! Trot come from behind board of the head. Yes. Fire a shot or two in that direction to cover me, Tonto. I'll go to the woods and try to surprise him. Hidden by the heavy brush, the Lone Ranger cautiously made his way forward until he was opposite the boulders. Seeing a figure crouched behind them, he called out... Reach you and don't move! Wait, don't shoot. I'll drop my gun. Oh, boy. How old are you? Nineteen. Who's with you? No one. I saw you wearing that mask coming along the trail with an Indian. I knew you were the man who led the attack in our wagons this morning. What? I suppose you'll kill me like you did my older brother and the other men.  I don't understand what you say about attacking your wagons this morning. You understand, all right. I saw you.  I'm the only one who escaped.  Well, don't hold on, son. Someone else led that attack, not I. Believe me, I'm not an outlaw and I don't intend to harm you. What's your name? Gary Biggs. Gary, I want to help you. My Indian friend is waiting back there. Will you come with me and tell us everything that happened this morning? Well, I... I'll host to my gun. Pick up yours, Gary. Then we'll go meet Toto and talk things over. The Lone Ranger and Gary joined Toto. Then they camped in a clearing nearby. When the youth finally realized the masked man and Indian were not going to harm him, he talked freely. I was coming west with my older brother, Frank. He was in charge of the four wagons. Yes. Just after dawn this morning, I... I was riding behind the wagons when Indians, led by a masked man on a white horse, came over a rise, yelling and shooting. I see. I... I jumped from a horse and ran into a grove where I hid in a thick brush. When the fight ended, my brother and the others were dead. Wagons burned. Oh, I'm sorry. Indians took the horses all except mine. I found him back in the woods. Gary, we're terribly sorry about what happened. If you'll trust us, we'll take care of you for a while. But... I'm all mixed up.  Listen, Gary, Toto and I came to this territory to find a notorious outlaw named Carl Kiel. He's a German and speaks with an accent. Oh, see. I remember hearing him shout notice. And he did talk strings like. Oh, me sure that Kiel, King of Bobby. Then see him wear a mask, ride white horse. Yes, Toto. Gary, while you have food and get some needed rest, Tonto and I'll ride to where the attack took place. Perhaps we'll be able to pick up Kiel's trail from there. After Gary had eaten and had been made comfortable in a lean-to, the low ranger and Toto left the camp and followed Gary's back trail toward the scene of the attack. As they rode over a rise, Toto pointed and spoke. Look, King of Bobby, many men dig a near-burned wagon. Those are, those are. Must be men from nearby ranches making grays for the victims of the attack. Ah. But then spoiled trail left by one who attacked wagon. Yes. We'll leave before we're seen. Come on, Toto. Get him out of this car. Toto, we'll return to camp and see how Gary is. Perhaps this afternoon you might hear some news of Kiel and Dove Creek. Huh? Take Gary with you and have him report the attack to the sheriff. Huh. All right, let's hurry. Come on, Toto. Get him out of this car. That afternoon, Toto and Gary Biggs rode to town to the sheriff's office where Gary told of the attack. The next day, in a secluded cabin a few miles back in the hills, Carl Kiel looked up as a fierce-looking Indian entered. Hmm. Well to go. You bring news perhaps? Huh? Me scout, many miles east. See, other wagon train come west. It may be reached plain where trail cross in morning. Oh. Ha-ha. Ah, that is good, Toto. Very good. It was a smart idea to join your gang of ten renegade braves with my eight men. Neither group alone was strong enough to attack the wagons. It was also smart to have my men disguise themselves as Indians. It's more confusing and terrifying to the pioneers we attack. He'll give the wagon train you saw a warm welcome to go. Braves, be ready. You give horses and guns like before. Yeah, yeah, yeah. Your Indians get all the horses and guns, and my men and I are to take whatever else day is worth taking. Ah. My, uh, my men are already at your camp, so tell them we attack in the morning. And later you and I will meet here and make definite plans. That morning, the lone ranger and Toto again tried to find Kiel's trail. Gary Biggs, who stayed at the camp, decided to take a ride. He rode into the hills and finally stopped on a ridge overlooking a small valley where the Indians were camped. Hold it, hold it, hold it. Oh, God. An Indian camp in that valley. Maybe they're the ones who attacked our wagons. The youth crouched behind the brush as he paused to survey the camp below. Suddenly... He ought not move. What? He may have knifed it back. Hey, that knife. Take it away. No, no, no, no. Me have knifed it back. Hey, that knife. Take it away. Why you spy on brave? I'm not spying. Get not good. You'll get on horse. Come with Tugo. Now, wait a minute. He said get on horse. All right. Don't use that knife. Oh, look, I wasn't spying. I just saw... You will not talk. We take it to pale face, boss, man. Get him, get him. Soon, Gary and Tugo reached Kiel's cabin. The Indian shoved the youth ahead of him, saying... You're going to cabin. What's this you bring here? Him spy on Indian camp in valley. Oh. Tie him up. After we find out who sent him to spy on the Indian camp, you'll get rid of him. Tugo tied Gary securely. Then Carl questioned him carefully and at great length. Gary, fearing for his life, told of events up to the time Tugo had surprised him on the ridge. Finally, Carl turned to Tugo, saying... He tells of staying in camp with two men. A mask man in an Indian who are searching for me to go. Me not savvy. No, but I do. The men he speaks of have looked for me before. And because of them, my friend, I left the Pecos territory. It's not good. They're so close. You're right. The mask one is the lone ranger. Be here of mask one who'll ride big white stallion. Him plenty bad medicine. Yeah. But he and his friend the Indian will stop a bullet like any other man. Perhaps it will be better if we get to them before they find us. We'll leave this young man here and go to the place on the ridge where you found him. Why we go there? To follow his trail back to the mask man's camp. They will not be expecting us to hunt for them, Tugo. So we'll have the chance to shoot them from ambush. When the lone ranger and Toto returned to their camp, they found that the youth had left. His horse gone Kimosabe, and him not lean to. A young man his age gets restless, Toto. He must have gone for a ride. Him come back soon, maybe. Gary's been through a lot for one so young. So we're warning us of danger. Him look to woods to write. We're lucky though nothing's wrong. Walk with me toward those boulders over there. The mask man and Indian walk nonchalantly toward the boulders. Then as they suddenly sprang behind them. The warning saved us from being ambushed. We return the fire. Think they're two of them. Yes. I am right away. Through wood. We'll follow them. Come on. Easy, sir. It's not easy. Come on. Come on. Carl Keele and Tuko rode through the woods and over a hill to the main trail. Later as they turned onto a branch trail, Tuko spoke. You think them follow us? Yes. We must stay well ahead of them, Tuko. We'll go to the Indian camp in the valley. Then if the mask man and his friend appear, they'll ride into trouble. Get up there. We'll continue our lone ranger adventure in just a moment. He can capture outlaws cause he knows. He's got go power from Cheerios. Yes. He's got go power from Cheerios. Sure. Cheerios. The cereal that's fun to eat because it's shaped like little letter O's. The only ready-to-eat oat cereal with this fresh toasted oat flavor. And listen. Every delicious spoonful of Cheerios and milk is real muscle-building food. Each spoonful contains vitamins, minerals and proteins your body needs. Yes. The good things in a Cheerios breakfast do good things for your body. Help you have healthy nerves, good red blood, strong bones and muscles. Yes. Cheerios is made to give you real go power. So every morning, get going and keep going with Cheerios. Then you'll hear people say, He's feeling his Cheerios. Now to continue. As the lone ranger and tato reach the main trail, they saw the two outlaws turn onto a branch trail and disappear over a rise. At that moment, the masked man and Indian saw the sheriff and posse approaching. They stopped. And after the lone ranger identified himself, told about Keele and Togo, then joined by the posse, they continued after the two men. The delaying of the lone ranger and tato allowed Carl and Togo to reach the valley and arouse the Indians. After telling them to capture and torture the two men who might be following him and Togo, Carl spoke to Togo.  Why you want young brave here? I want him to watch while his masked friend and the Indian burn at the stake. Then him burn his sacrifice to thunderbird spirit. Hey Togo, by the time you get back, we'll have the masked man and his Indian friend as captives. The lone ranger and the posse followed the tracks left by the horses of the German and the renegade. As they near the entrance to the valley, Tato signaled a halt. Kim Sabi. Yes? That friendly sign of Indian near. Oh, I've noticed Tato. Will we do? We'll have Tato scout ahead before we go any further. Not a good idea. He's just got his equivalent. We go on foot, not belong. Tato disappeared into the sea to brush like a shadow. In about ten minutes he returned. Here comes Tato now. Kim Sabi. Yes? It keeps good when we go on ahead of posse. What did you find out? Braves and war paint camp in small valley ahead. What? We not see any squaws or dogs. That means it's a war party. Ah, about funny brave. They must be the Indians, Keel, lead and a wagon attack. No doubt about it. We got eighteen men. If we can take those renegades by surprise. I'm sure we can, Sheriff. Dip it in. Yes, yes. Take half a minute and follow Tato along the ridge to the other end of the valley. Then move him that way. All right, Sheriff. Stay there until you get a signal of free shots in quick succession. Right. How do you and the deputy keep a sharp lookout? If you're discovered, you give us the signal. Then ride into the valley from the far end shooting. We'll wait here and keep them from escaping at this end of the valley. Ah, let's go, Tato. He's here. He's here.      At the Indian camp, Carl waited in one of the wigwams and watched through the opening, as the braves mounted mustangs and rode toward the valley entrance, through which the lone ranger and Tato were expected to approach. That masked man and his Indian friend will get a reception they didn't expect. I'm sure they'll follow our tail to this camp. Carl was suddenly startled when... But the shots were followed by the yipping of the posseman, as they rode into the valley from both ends. Stepping quickly from the wigwam, Carl took in the situation at a glance. The Indians are trapped. The masked man has brought others with him. I'll try to escape up the slope. Carl ran around the wigwam to his horse.   Come on. He headed toward the steep slope at the side of the valley, hoping to escape during the excitement, but the lone ranger saw him, and leaving Tato and the posse to subdue the renegades, he started after Keele. Come on, Keele! After Carl reached the brush-covered slope, he turned and looked back. The masked man is after me. Carl's aim was affected by his panic and by the movements of his horse, and his bullet went wild. The lone ranger, aware of the situation, withheld his fire and bent low over his great stallion's neck as he urged him onward. Faster, big fella! Faster! As silver increased his speed, the fleeing outlaw once more turned and fired. This gun must be empty now. Come on to the land! The gap that separated the lone ranger from Carl's steadily closed. Then silver moved alongside the outlaw's horse, and with a mighty lunge, the lone ranger sprang toward Carl. Get off the horse! He dragged the outlaw from the saddle. Both men were well-built and muscular, and as they struck the ground, Carl landed on top, momentarily having the advantage. He raised his empty gun over his head. This, my friend, will break your skull! The lone ranger grabbed Carl's upraised arm, and the veins stood off for a moment on the necks and arms of both men as they struggled. With a sudden lunge, the lone ranger rolled on top of Carl and pinned the German strong arms to the ground. For a moment, he stared down into Carl's hate-filled eyes. Then the masked man suddenly loosened his grip on one arm and swung a heavy blow to the outlaw's jaw. This should do it! Get to your feet! Pulling the stunned outlaw to his feet, the lone ranger quickly used his lariat to tie Carl securely. I will hold you, Keele. The posse is to build your Indian friends. I'll take you down there and turn you over to the sheriff. With Silver walking behind him, the masked man approached the posse with his prisoner, and the men greeted him with enthusiasm. Easy, sir, easy. Sheriff, this is Carl Keele. I'm glad you got him. I saw you fighting up on the slope when I was a little worried. You will not have to worry, Sheriff. So I see. We learned from Brave that Indian named Tugo go to bring Gary here. We find out Keele and Indian catch Gary this morning. Take him to Keele's hideout. How do you know he's Gary? Keele tell Indians him to capture young fella who escaped from wagon attack. Then it must be Gary they captured him. Keele, where is that hideout? Speak up, where is it? A mile to the south on the branch trail. Sheriff, thought I'd now go find that Indian Tugo and Gary Biggs. We'll meet you at the junction of the main trail on the branch trail, where we turned off to come here. All right, we'll see you soon. Half an hour later, the sheriff and the possemen heard hooves approaching the junction where they were waiting. Sure. Here comes the masked man, his Indian friend. Yep. Well, Kelly, they have the young man and an Indian with him. Sheriff, this is the Indian Tugo who rode with Carl Keele. We met him heading for the valley with Gary Biggs. Did you hear any trouble? Tugo saw him coming. He was fixing the fire. When the masked man shot the gun right out of his hand. You were right, young fella. Oh, yes, Sheriff.  Keele's a mean hombre, sir. By the way, Mr. Hampton, after you and Tonto left, we found out something interesting. Oh, what, Sheriff? Eight of those prisoners aren't Indians at all. Their white men disguised as Redskins. Must have been the members of Keele's gang. Yes, and that explains something that puzzled me. I couldn't figure out why a band of Indians would need Keele to lead them in the attacks on the wagons. The fact that Keele's men are with them clears up that point. Without them to help, the Indians didn't have enough men to risk an attack. That's about the size of it, mister. Keele and his men will hang for the killings. Tugo and his braves will be punished by the army. Baby, we're young fella. What do you intend to do from now on? Ride with the masked man and Tonto? I'll answer that, Sheriff. Gary wants to stay in Dove Creek if he finds a home. Well, in that case, he'll live with me and my wife. We'll be glad to have him. I'll teach him to be a good lawman. I'd sure like that, Sheriff. Oh, that's fine. Sheriff, I'm sure you and your men can handle things now. Tonto and I will stop at the army post and request that they send troopers to get those renegade Indians. Good. Those Redskins are through making trouble around here. Adios, Sheriff. Adios, everybody. Hello, sir. We'll stop to see you now, and then, Gary. Let's go, Tonto. Come on, sir. We'll give in. There we go. But they won't forget you, I'm sure of that. They're the finest men I ever met, Sheriff. Believe me, I'll never forget the Lone Ranger. We'll return in just a moment for a word about our next exciting Lone Ranger adventure. Dead Man. Here's big news. Now you can get auto emblem models free of extra cost with Wheaties. You know auto emblems are the handsome insignia or name plates that identify one automobile from another. Some automobiles have their emblems on the horn button, others on the trunk of the car, or up front by the grill. Wheaties emblems are beautiful steel models of these actual auto emblems. Each emblem is a little bigger around than a donut. The brilliantly colored design is raised, not just painted on, and it won't break, it's made of solid steel. Eight different emblems are offered free with Wheaties, one to a package. There's Nash, Packard, Studebaker, Hudson, DeSoto, Chrysler, Plymouth, and Dodge. Here's how you get them. At your grocers, look for special Wheaties packages that picture an auto emblem on the front. The picture tells you what emblem is inside. The emblem inside is yours free of extra cost with a special Wheaties package. Hurry to your grocers now. Get your Wheaties auto emblems while supplies last. To save Tatl's wife, the Balone Ranger raced against time and a gang of killers determined to murder the Indian and the Masked Champion of Justice. Be sure to listen. The Balone Ranger, a copyrighted feature of the Balone Ranger Incorporated, is created by George W. Trendle, produced by Trendle Campbell Muir Incorporated and directed by Fred Flower Day. Tonight's drama was written by Dan Beattie. The part of the Balone Ranger is played by Brace Beamer, your announcer, Fred Foy. The Balone Ranger is brought to you by General Mills every Monday, Wednesday and Friday at this same time. Be sure to listen. This program has come to you from Detroit. This is ABC Radio Network.


Good morning, everyone. I hope you had fun last night So today's topic going to be about what goes down the drain I'll introduce myself quickly. My name is Yanir Quinn. I'm working in Red Hat since 2016. This is my first DEF CONF. So very excited. I've been working on several projects like Overt, Qvert, OpenShift and some more. Okay, so let's get to the topic. On our Kubernetes cluster, when you want to manage it, from time to time, we want to perform maintenance operations. Maintenance operation in our, let's call it, our synonym here is no drain. So to understand what is no drain, why do we need it and how we perform will go through three, the three questions. So what we're close? Are we actually draining for our Kubernetes nodes? Why do we do that? The cause and and how? In the how section will go through one or two Kubernetes APIs for node eviction. Understand the complete complementary concept of pod disruption budgets, and if we'll have enough time, I'll dig deeper in the concept of server-side drain. So we have our Kubernetes cluster with it, which is managing and orchestrating our containerized application, and it's considered to be fault-tolerant. One misconception might be that, okay, if I have enough free resources and something goes down, my node goes down, the Kubernetes scheduler would just, okay, we'll make sure that the pods will be scheduled on another node, maybe autoscale or downscale our cluster, and everything is fine and dandy. But that's not always the case, and it's pretty much a misconception, and to understand that we'll dig a bit deeper about the reasons and the consequences we have. So we'll start with what is being drained from the node. So we have our pods containing our containers, which are running applications, microservices, and processes, some profound examples for our world, our virtual machines running on pods, as we do in Cupert, storage pods, like in OCS, and these are the heavy workloads. Okay, so I want to talk about what are the reasons for for maintenance. Before that, let's understand how pods are being evacuated or dropped for a specific node. Either intentionally or not intentionally. So we can split it to two by disruptions. So we have involuntary disruptions. These involuntary disruptions happen without us intentionally doing something, or by us or by something. Some example is the classic one, hardware failure. Well, simply my machine breaks and goes down, and my node is down. Human error happens from time to time. Let's say a cluster, I mean accidentally brought down a VM, and now I don't have any node. A VM can disappear on my cloud or hypervise on our cloud. Kernel panics, of course. A node can go missing if I have a network partitioning, and it just disappears. All of these are classic day-to-day failures that happen not only in a Kubernetes cluster, but in your old, all other clusters that you know, even 10, maybe 20 years ago. Another thing is node going out of resources. I left this to the end because Kubernetes has a mechanism to handle that. So we might want to prevent these things from happening if we can't evict these workloads running on our nodes and are gone because of involuntary disruptions. Some preventive methods can be defining for a pod what actually resources it needs so our node won't go out of resources. Of course, using high availability. We can replicate our applications across the Kubernetes nodes. Going even further, you can spread the application and make it even more high available using anti-affinity and multi-zone clusters. So one of the examples I talked about for preventing out of resources is defining in a pod how much memory or CPU I want a container to use. In this example, we can see two definitions of limits and requests. So we don't want a container to cross a certain limit and cross the pod. If a container crosses its limits, it will probably get restarted or get a runtime error. We can allow in a pod to have a certain limit if we know we'll have bursts or spikes. And usually, it will have, is usually a request of memory for day-to-day actions. So this is one of the preventive methods. Now we're going to the most more interesting part of our talk is voluntary disruptions. So this can be caused by either the application owner or the cluster owner. Starting with application owner, we can just delete a deployment or a controller that manages the pods and then the pods can just disappear. If we update a pod template, make some changes in the spec, it might cause a restart and the pod will be erected and then recreated by Kubernetes. Also, directly deleting a pod, although it might be by accident, can be caused a voluntary disruption. Now for cluster administrator, and this comes down to maintenance. I can decide I'm a cluster administrator, I now want to perform a role in update, a kernel upgrade, hardware replacement, but I know I'm going to do that and I want to prepare for that. In order to do that, I will want to do it gracefully and allow my pods enough time to terminate. Another example for such maintenance operation, voluntary disruption, is draining an old from a cluster to scale down the cluster. We might also want to remove some pods from time to time to allow something else to fit on the nodes. But let's focus on the top two. So this is why and also what?  So draining operation is comprised for two main actions. One is codoning the node. Codoning means I shut the node down, I don't, sorry, I have a barrier on the node. I don't want to allow any new pods to be scheduled on the node because I'm going to eventually shut it down. And the complementary action of the node draining is evicting or deleting the pods if eviction is not possible. It's also possible to prepare for node eviction and just create, call kubectl, coordinate node before we actually want to start the eviction process. Okay, so the CLI for draining a node is just kubectl drain with the node name. The node becomes unschedulable. You can see it also presented as a field or also attained. It tries after the node is condoned to evict all pods from that are running on that node. If it can evict these pods, there are some exemptions which I'll show soon. And if eviction is not supported, Kubernetes just calls the delete API. So let's start with the delete API. You can call it delete API and Kubernetes node to just delete the pods. The delete API also gives you a field which allows you to gracefully remove the pod, meaning I'll define a certain timeout before it picks out the pods along some workloads or some processes to finish, to complete. When we have involuntary disruptions, for example, if my node suddenly breaks down, we don't really have a chance to have this grace period. So we might have some sort of chance to get it in before we see a node comes down. Again, this is only for involuntary disruptions. So now we'll talk about the eviction API. Eviction API came to us to Kubernetes 1.7 and above. And we use it instead of directly deleting a pod. Here we can also avoid calling an external command. We can utilize this API in our code base or project if we want to. And we have also finer control over the eviction process. So you can see it pretty much looks the same like the delete API with a special kind for eviction and another concept that coincides with this eviction API. That is pod disruption budget. So what is pod disruption budget? It limits the amounts of pod that can be evicted by the eviction API to a certain limit. By definition, you can put on the custom resource of pod disruption budget. Some examples. For example, I have a stateless front end, running UI operations. And I want to keep the capacity at 10 percent. I want to keep the capacity that won't reduce more than 10 percent. So I define a pod disruption budget with a parameter called minimal available pods at 90 percent. Minimal available, for example, can be also a number of pods. Percentage makes it more easier to work in high amounts. In a single instant stateful application, I want to block eviction completely unless I tell it to proceed. I want to have control over that. To do so, I can just define max unavailable zero. That means I don't want any pods to be evicted from the cluster, from that node specifically. Unless I remove this pod disruption budget, I remove this limitation. And then the eviction API can go on and evict that pod. We also have a quorum example, like using ETCD in a multi-instance cluster, where I can actually utilize this pod disruption budget to define my quorum, setting a number for max unavailable for one, or defining minimum available pods that should stay on that node for, let's say, three out of five to maintain the quorum. So this is how the pod disruption budget CR looks. It's quite simple. You have in the spec the parameters I mentioned and their value. And a matching label to the pods I want to associate the pod disruption budget. When trying to evict and the pod disruption budget exists on the pods, I can have several scenarios. Either it's granted, pod disruption budget doesn't block me from evicting my pods from the node. So the eviction API just goes through and we're all good. Pod disruption budget is not accepted. It's not respected. So it will block eviction and you will get an error. This is 429, which is a bit weird for me. Or either I have a misconfiguration assigning too many pod, multiple budget to the same pods and that resulting in an error. So sometimes that can result in a broken state. Let's say I have a controller. This controller tries to evict pods using the eviction API. And I'm getting blocked by pod disruption budget. Or either my replacement pod is not ready. Let's say I terminated the pod. I need it to be rescheduled somewhere. And if it didn't reschedule somewhere, the operation did not complete in my controller. Handling these solutions can be by pausing the controller's work or deleting the pod actually manually or forcefully after a long time. So I was talking about the deletion API as part of kubectl drain. So we have some special pods that won't be deleted immediately. One permanent example is demon sets, which is not that horrible to delete from a node because they will spin up anyway. But in order not to block the eviction, you can specify a special flag to ignore demon sets and just go on with that. Mirror pods won't get deleted that are mere pods are representing static pods on the node. Unreal replicated pods, these are pods that don't have replica sets and deployment and they will not won't spin up because they don't have such deployments or replications. So if I'm sure it's okay to delete a pod from the cluster and evicted without it getting replicated, I can use a force flag for the kubectl drain command. Also pods with empty deers, if we're certain that it's okay to delete or evict them, we can just use the delete local data flag and it will be deleted. Okay, so when coming out of just simple application running on pods, we have some heavy workloads. Formulate exception examples are virtual machines. Let's say I want to perform an upgrade on a node or maintenance operation and I don't want my VMs to go down. Part of the classic process of that is migrating my VM. So I call a VM migration by initiating a drain command. I don't want the VM just to go down. So same thing goes also for storage nodes. I don't want operation of draining and eviction to be complete unless my storage process was completed, meaning replication, rebuilding of storage data. These are also very long and heavy transactions. So to deal with that, we want to allow or utilize pod destruction budget with some sort of mechanism. We want to put a pod destruction budget on the pod running on the node. We would like to signal a controller that will accompany these processes to see if eviction was successful, if the complementary action of rebuilding or replicating storage nodes or migrating a VM was completed before lifting the pod destruction budget and allowing the pods to finally be evicted. So this is one of the examples I was talking about. I have here a free machine cluster which I can allow only one machine to go down or one node to go down and I would like to maintain it. So we have our pod disruption budget controller that looks over all of our nodes and let's say I'm trying to evict one node. So first you will want to see that the process was ended. The replication of data was, for example, ended. After you saw that it will remove the pod destruction budget of maximum unavailable zero and will allow these pods to be evicted. Another scenario. I want to evict two nodes at a time but we already started one process. So we also observed that. I'll mention again that I want to signal the process to begin like VM migration or replication but how will I know that? So we can use canary reports for that. I saw a canary pod getting evicted. I will start some process to go on migration or replication. I saw a tint on a node added so I'll start the process on migration or replication. So all of these processes can be initiated or usually are initiated by Cube CTL drain command which is client side command. But what happens if I would like something extra, something more? I want the server side drain meaning I just create some CRD or call and I get some feedback. For example, if I want to use a UI to start server side, to start draining my nodes, I don't really have an option right now using the CLI command. So server side drain will be great for you. I want to accompany the process. I want to see what events are going on. If the maintenance on drain in operation was completed or I have some errors. Automation. For example, I have a machine configuration. I want to change and I want to allow the process to be as automated as possible. So entering some drain commands or drain CRDs for that process will be helpful. There are some several solutions or semi solutions currently in Kubernetes that you might take a library, put it aside. But then you get not only one solution but 10 or 20. So one solution for draining for server side drain would reduce the use of multiple solutions. So that basically will look the same. I see I have one more minute. But I want to show you a quick demo of something we wrote in Kubert for server side drain. It will be really quick. So one less question but I think it's better. So this is an example of a controller I wrote utilizing server side drain. So I have a cluster with one node on it. Sorry. We have our controller which we call node maintenance operator which is running on node one. This is a cluster of two nodes. And now I see all the pods running on node number two. And now to create a server side drain I'm using a CR which is for node maintenance. So in that CR I just give it a general name. The node I want to evict and the reason for that. Of course later on you can add some more fields like status and events. So I invoke this CR. The controller has detected that I created a CR for that. It already called on the node. As you can see down there that the node is not scheduled. And we can see node two has been called on and scheduling is disabled. And also most of the pods have been evicted except the demon sets that are running here on the nodes. Also taking a look at the controller logs you can see all the pods that could be evicted were evicted. So ending that process or returning to node, switching the node back up we just delete that CR for the specific node. And the node becomes schedulable again. And all the nodes are ready. So that was a very simple example but it was a bit fast forward. A lot of processes happened in the background. Nodes were evicted. Before that the node was called on. All of our workloads migrated or were rescheduled on different nodes. And that's a nice example for server side drain. If you want to know more I would be happy to share some links afterwards on the slide for the node maintenance operator. So yeah. So now I have three minutes for questions if you guys have any. Should have made the demo longer. Okay. So the question was if I can install these features by CRD or operator. I'll refine that. This specific feature exists as an operator. It's encapsulated as an operator but it's basically a controller which has its custom CRDs. Once you deploy this operator let's say in an open shift environment you will have the custom CRDs. You can just create the specific CR for draining the node and the controller will detect that and just start the process. Yes. Good question. So the question is if it's our own controller and where do we stand on the Kubernetes side. So we wrote this controller out of a need because it wasn't server side drain didn't exist in Kubernetes. There is a Kubernetes enhancement process. By the way if someone here is from SIG node come talk to me later. An enhancement procedure on Kubernetes to get it accepted and make life much easier for everyone if we have this type of operator. It doesn't necessarily have to be one to one but the concept is the same concept. So a cap is pending. It's getting attention but in more use cases but I think that is currently the stage right now for server side drain in Kubernetes. They are not too eager to push it forward at the moment but we keep bugging them. Another question. Okay thank you guys. Thank you very much.