A fiery horse with the speed of light, a cloud of dust, and a hearty hyoselver, the Lone Ranger. With his faithful Indian companion, Tonto, the masked rider of the plains led the fight for law and order in the early western United States. The stories of his strength and courage, his daring and resourcefulness, have come down to us through the generations. And nowhere in the pages of history can one find a greater champion of justice. Return with us now to those thrilling days of yesteryear. From out of the past come the thundering hoofbeats of a great horse, Silver. The Lone Ranger rides again. Come on, Silver! We're heading for Beacon City! I'll Silver! As the Lone Ranger and Tonto approached the town of Beacon City in the heart of the cattle country, they saw a strange sight. Thousands of sheets of paper tossed on the breeze came toward them like a snowstorm. The masked man wheeled Silver to one side and snatched one of the large squares out of the air. Rain up, Tonto. See what this is. Oh, there's Silver. What's happening in Beacon City? A whole edition of Bob Turner's newspaper seems to be flying around the wind. Oh, that newspaper? Yes. This is the front page of the Beacon City Star. Oh, we know Bob Turner. Meet him long time ago. He's one of the most honest pioneers in this part of the country. I wonder if... I don't know. Oh, what matter? This paper has an editorial on the front page. Turner has fired both barrels. What do you say? The third sheriff has been killed since the first of the year, shot in the back. Turner's demanding that the people organize and throw out the crooks that are trying to run Beacon City. And that's why paper fly free on wind. Looks as if something has happened. Come on, hello. Let's find out. Get him up, scowl. Go on, Silver. The office of the Beacon City Star was one of wild disorder as ace-wills and two rough companions swung heavy axes right and left. The big glass window had been shattered and the jagged bits of glass were strewn about the floor and furniture. The three men in the place held heavy axes and with these they concentrated upon destruction of everything the office held. The valuable printing press for giving brought from the east of great expense was already broken beyond repair. But one of the men continued to smash the broken parts to increase the ruin. Chairs, tables and other pieces of furniture were being smashed in the kindling wood. The large filing cabinets of letters were smashed open and the contents were scattered. Wills saw the neatly arranged type and rose against one wall. He shouted... Smell that case of type. Jump it over so it's all mixed up. That's it. Now rip up a supply of paper. We didn't fix it so he won't print a newspaper again for a long time. There's a type all mixed up for plenty. Now smash your cabinet while I'm at it. What's going on in here? Get that mask, man. All right. Don't get them. Please fix them. The danger in Toronto charged. Toronto drove a hard fist in the face of the first man. The salmon sprawling across the floor. The masked man dodged a blow by Wills and jabbed the leader in the stomach. The blackers landed a blow on the back of the Lone Ranger's head that stunned him momentarily. But Toronto not bleak to one side long enough the Lone Ranger regained his balance. From the floor, Bleak drew his gun. He was just about to fire when the Lone Ranger Wills snatched his own gun from his holster. My hand, my hand. No more gunplay. Don't hit me again.  Oh, my hand. My hand is broken. You hit the wall and you missed me.  Get in my hand, Toronto. I need to lift you up. Oh!  Keep an eye on those others when I talk to this one, Toronto. No, wait. You've got no right. I know you. You're Ace Wills. You're on the cafe. Well, what of it? I saw the last copy of the Beacon City Star. You didn't like it. No care, stranger. We can handle things in Beacon City without your interference. I don't like the way you handle them. Where's Bob Turner? None of your business. And you'll pay for hitting me. Where is Turner? I said it was done. Oh, let go of me. He's home. He's home at his house. Where is his house? About a half mile to the south. White house with the green shutters. Wills, you're going to settle for the damage you've done here as soon as Turner figures the cost of putting the newspaper back in business. If you've injured Turner, you're also going to pay for that. Yeah. Well, we'll see who does the paying. Make your friends and get out. Huh? I'll get square with you. Get out. All right, all right.  I'm going, but I'll get square for a busted hand. You two just wait. I don't turn you back to the windows, Tonto. Those dogs will shoot you in the back. That's right. You certainly made a mess of this newspaper office. I wonder if the press can be repaired. It's badly smashed. Oh, and this happened because Turner tell truth. Turner, you've got to get to his house and see if he's hurt. Come along, Tonto. Yes. First man has ever dared fight with ace wills. Ace wills. Ace wills. I hate to be in their shoes. I hate to be in their shoes. Hey. Hey. Better get out of here, fellas. Folks, don't do them things to the ace wills. All right, mount up, Tonto. Well, a helly posty, Turner. Yes, but we'll keep our horses with us.  The lone ranger remembered Tom Turner from an earlier acquaintance. He knew that the publisher was utterly fearless, daring and determined. He knew that he was a firm believer in the freedom of the press and the fact that lawlessness must be stamped out. He knew that Turner must be badly beaten by the hoodlums and realized that the destruction of the newspaper would be a far greater blow than any physical damage that might have been done. His one hope was that Turner was still willing to fight. Is he and Tonto reigned up in front of the house? You hear what crowds say. Yes, they seem to think that wills can do whatever he wants to. It's a bad situation for any town. I hope Turner's not hurt. Nobody near his house. After this, I don't suppose anyone will dare be seen talking to him. Oh, fellow steady. You better stay here with the horses. He keep close watch. Keep a sharp watch with a solid wall behind your back. Mask man, do come in. How's Bob? He's licked. He's in the next room. You're his wife? Yes, I'm Mary Turner. Bob saw you right up to the office and go in. Then he saw Willis and his friends come out. Bob, he's done it. He's finished me. Yeah, let me take a look at you. Hmm. There's a bad bruise on your head. No, it's nothing. I don't mind that. It's my newspaper. How badly is the office wrecked? Well, it's going to take a lot of work and a lot of money to get your paper going again. I can do it. Willis won't let me. I'm through. You're not through, Turner. Uh, what's the use? When a gang of crooked-cut throats can do what Willis and his friends have done to a town, there's no place for an honest man. And we've got to make a place. Now, uh, you've made the start. We'll go on from there. Hey, uh, I saw you. Yes, Mary said you did. You'll have to get out of town, friend. They'll shoot you the same as they did those three shares. I won't turn my back to them. Now, uh, look here, Bob. I must be some honest men in Beacon City. Sure, they're, uh, most of them are honest. But they can't fight a gang like Willis has. Willis wouldn't have a gang of outlaws. That is, uh, unless he used them for something. What's his game? Cattle wrestling. Oh? Beacon City's a stopover place in the trail to the North Country. Willis gets friendly with the herd drivers that come into the café. He learns where cattle are being moved. Then his men go and get the longhorns. Where do they take the stolen cattle? Well, nobody knows except the gang. I see. They've got to hide out somewhere. But there's no telling where it is. Uh, there's no use. You can't do anything. Willis is too strong. You'll only get killed for trying. So we'll see. Bob, uh, who is the sheriff now? Well, Will's appointed an old galoot called Tumbleweed. Yes, he'll do anything Willis tells him, not with a hoot. I see. Uh, where can I find him? Well, if he's not sprawled over a table in the café, he'll be in his office. All right, then that'll be my first stop. But wait, you can't. Bob, uh, Bob, can you round up the honest men quietly and secretly? I might be able to, but... Uh, you know who they are? I guess so. Well, then get them ready for a fight. Uh, can you count on them to fight? If there's anything to fight for.  They'd fight for it and fight hard if there was any chance of winning. Get them together. Tell them that there is a chance of winning. But what are you going to do? I, uh, I'm going to call on the sheriff. And I'm going to make a few arrests. That evening found Will still raging over the manor in which the masked man had handled him. In his café, he found it hard to mask his feelings with a fixed smile and a congenial manor. One of his men whispered to him and he sauntered to a table where a white-haired man was seated. Well, stranger, tell me you're driving your herd north. I reckon they told you right. My name is Will's. This is my place here. Yeah, it's first raid. I always make it a practice to stand and treat for a man that's handling cattle. I figure a man thinks enough of my place to stop off overnight. Well, at least I can do this by a drink. Now, that's downright nice of you, Mr. Will. Help yourself. From now on, it's on the house. Well, this is something first raid.  Well, that takes some handling. Yep, sure does. Well, here's how. Drink up. You, uh, you want to be careful. There's been a pack of outdoors operating around here. Well, we're watching out. The last time I heard it was put through Eagle Pass, it was raided. That's a bad piece of land there. It is, eh? Yes. You can't see what's hidden on the rocks on the side of the pass. Your men could be picked off before they could fire a shot. That's something to know. Well, I thought I'd better tell you as long as you're a stranger in this country. We're trying to prevent any more cattle robberies. It, uh, it gives a section of bad name. Well, thanks, Mr. Will's, for telling me. You can skirt Eagle Pass if you want to. That'd be a good idea. Is your cattle grazing now? Yeah. Now, if you haven't put it to graze too far from here, well, I can show you how to avoid Eagle Pass. It ain't so far. Only about seven miles east on the range. How many men have you? I get about a dozen. Well, just a minute. I'll get a pencil and paper and show you how to avoid the pass. That'd be right. Friendly of you, Mr. Will's. Excuse me for a minute. Pete, twelve hundred ahead, seven miles east. It's on the range with about a dozen men. I serve it. You know what to do. I'll get the boys together and take care of that cattle tonight. It can depend on me, boss. So long. Hey, Mr. Will's! Mr. Will's! Hey, what's the matter, Tumbleweed? No, it's the deputy. It's the deputy sheriff. What? I just got away to get here. He'd come into my office a while ago, wearing a mask. Mask? Will you get it? Yeah, I hear. Go on, Tumbleweed. What did he do? Well, he made me swear him in as a deputy and give him a badge. I didn't have no choice. I had to do it. Mask and all. Well, we've been looking all over for him. Look here, Tumbleweed. No, no, let me go, Mr. Will's. I couldn't help myself. And I think you need a deputy. I'll tell you who to appoint. I know, but... He couldn't help it. Well, I couldn't, honest. I couldn't. I don't know why he wants the badge, but then he wanted it. Hey, boss, there he is. The mask man here. There's the critter. All right. You're all covered. I'll get him. Come on, then. Leave your guns where they are or someone will get hurt. He's got a dozen men with him. I'm backed by men of this town who want to see law and order. The name of the law we're putting Will's, Bleak and Jake Hastings under arrest.  They're just the crazy men among us, Will's. Oh, you haven't had enough yet. Will, turn to the next time you're... Do you have anything to say, Will's? You can say it in court. Take their guns away. You won't keep me until long. And when I'm out, this town will be too small for you and every man that's with you. Tumbleweed will take the job of guarding these three. But, but, but... Ah, hold on. Just what am I arrested for? You didn't ask that, Will's. The three of you erect a newspaper office. There's a law against that sort of thing. And that law applies to you, the same as it would to anyone else. Now, get going. The curtain falls on the first act of our Lone Ranger story. Before the next exciting scenes, please permit us to pause for just a few moments. Now to continue our story. Ace Will's and one of his friends were in the small stone jail. And Bleak, the other henchman, was in the cell adjoining the sheriff's office. After his fury subsided, the cold logic that was part of Will's makeup became dominant. Take it easy, Jake. We'll be out of here in the morning. Well, the idea of it, that mask man making himself a deputy sheriff. I won't hold that against old Tumbleweed. He had to swear the mask mania, and he had no choice. I know, but, but of all things, you and Bleak and me charged with smashing up the newspaper office. Sir, I don't think he got off mighty easy when he sees the next things that happened to him. Well, where's the rest of the gang? Why don't they come tear this jail apart? I sent Pete out to take him on a little job. Tonight? Yes. When Pete gets back in the morning, he'll hear about this. Then you'll bring the gang? Right. Well, why did they put the two of us here and Bleak over in the sheriff's office? I don't know. How far away does the gang have to go to take care of this job? Not very far. Cattle job? Yeah. That old timer I saw you talking to in your cafe? Yeah. He's got around 1,200 head and Grayson with around a dozen men to watch the herd. Pete and the boys won't have any trouble moving that livestock. He'll take care of it tonight. They're probably taking care of it, right? Pete and the gang of rustlers rode through the night toward an expansive plain where cattle grazed. There's the cattle, boys. See any of the riders? I don't see a sight of them. You know what to do when you do see them? Drill them. Right. Are we going to drive that stock to the usual hideout? That's right. Rain up. Rain up, boys. Now what? Boys will use the usual methods. Ride in hard with guns blazing. Stampede that herd, drive them to the hideout. Lefty, Tim, Kenny, you ride wing on the north. Rest of us will take care of this side. Anyone see any of the cowhands? No sight of them. Maybe they're on the other side of the critters. Maybe they figured the law of horns would take care of themselves, and they went to town for the night. If you see anyone, let them have lead. Ready now? Then let's get going. Get on the land! The band of outlaws charged at the herd with guns roaring. The outlaws were experts at starting a stampede. They knew exactly how to do it in a minimum of time. They charged the cattle, shouting wildly, and firing the guns at close range. In almost no time, the peacefully grazing herd was transformed into a wild mass of frantic, half-grazed, fear-ridden beasts whose sharp hooves pounded the ground in an increasing roar, trying to escape the men and guns they feared. The outlaws rode expertly, keeping pace with the charging waves of longhorns while guiding them toward the distant and well-hitting valley that had for some time been their headquarters. Hard-riding with the thieves kept the mass of longhorns packed in close on the sides. The men who had been on guard were nowhere to be seen. They might have snuck off to town. Maybe they got scared and let out. They weren't complained for not fighting them. No worry, Kim's stampede. Fast till they go, the sooner we get to the hideout. In the meantime, Turner and the lone ranger were in the sheriff's office with old tumbleweed. I don't know what's to come of me after this. Mr. Wills is going to be mighty mad at me. Wills is in jail, tumbleweed. As long as he's there, he can't hurt you. I know, but then... Wills and his waiter go to pay for every bit of damage they've done to my printing press in his paper office. Now, look, Turner, you know we won't never do that. You know how Wills has got this talent in the palm of his hand? What'll happen? Well, I'll tell you, in the morning, his men will hear about this. Maybe we'll jail these men. Yeah, but what's the charge? You can't prove anything again. We'll see. They'll get Wills out, and even if they don't, he'll go free when he goes on trial. He'll pay you fine. He'll pay it, then he'll start in to get it right back. You know as well as I do that his gang steals cattle. Oh, Dad ratted all, Turner. No one's one thing and proven it's another. You can't just say a man's a thief. You've got to prove it, and there's no proof. Turner, we'll clear out and let tumbleweed think it over. All right, what have you seen? Now, look here. Won't you please give me back that deputy badge and get clear of Bacon City? Well, when I'm finished here... there's going to come a long time before that. Keep an eye on your prisoner, tumbleweed. If he gets out, we'll hold you responsible. Oh, what am I into? Why did I ever take this job as sheriff? Hey, deputy, you wouldn't ask. Well... Look, there's room in the regular jail for three men. Why can't you put Bleak in with Ace and Jake? Why leave them here where I got to guard them? It'd be too crowded with three of them in one cell. Maybe around a couple of hours, tumbleweed, and take your place on guard so you can get some sleep. But, but... You wait, tumbleweed.  Maybe if you unlock this barred door, I'll talk Ace and they'll let you stay alive. I won't listen to you. Now, stop talking to me. We're going to get out anyway, tumbleweed. You know that, don't you? I don't know a thing. You keep me here and you'll be sorry for it. Hey, Ricky, you know what'll happen to you. I won't listen to you. I'm hanged if I'll listen. What can turn that masked man do if you let me out? Nothing. You can tell him I escaped. They won't shoot you for it. But you keep me here and you'll be shot sure as my name is Bleak. Stop talking and I'll tell you now, stop it. Why don't you try and make me stop?  You know that. Oh, so we've got company. Well, what do you want? I don't know as you know me. I'm sort of a stranger in town. What do you want here? Nothing, Mr. Sheriff. I just sent over some hot coffee for you. I reckon it'll taste real good. Well, I sure will. It sure enough will. I was coming this way and they asked me to fetch it to you. I've got to herd a cattle near town and I'm a takin' north there. That'll hold you. That was done neat. Hey, who are you? I'm just following orders, that's all. So you're talking to Wills in the cafe. That's right. Mr. Wills was right fine to me. He warned me about outlaws nearby. I reckon the masked man was in with him, huh? Yeah, yeah, sure. Anyhow, when I heard that Mr. Wills was jail, I figured I'd help him out. Just a second I'll have this here door unlocked. You won't be sorry for this. I've got the keys here. One second now, I have to find the right one. You ought to get to the hideout as fast as you can. Hideout? Yeah, if your horse is saddling, waiting out in back of this building. What about Wills and Jake? I was told to send you to the hideout as fast as you can get there. That's all I know. All right. There you are. Better get your gun and travel fast. All right. This is my gun and belt right here on the desk. I'll stay here and make sure the sheriff stays the way he is. If he starts showing any signs of getting conscious, wrap him on the head again. As soon as he left the sheriff's office, Bleak headed north to the hills. A narrow pass between sheer walls led into a wide, well-watered valley. Countless steers could be seen in the moonlight. And at one side there was a low house in which lanterns burned. Bleak joined the gang in the house. Bleak, where'd you come from? From the sheriff's office. I've been locked up there. Locked up? Yeah. Where's Wills and Jake? Are they here? No, the last I saw of Daisy was in his cafe. He gave me orders. What has you locked up for? My little fool's sheriff. Come with me. Yeah. He appointed a new deputy.  He was working with Turner. And he came into the cafe with a dozen or more men from town. They jugged the neighbors for rucking the newspaper office. You mean the bosses in jail? That's what I'm saying. But I thought he got out by this time. Not that we know of. We just got here with a new lot of cattle. How did you get out? An old cowman came and cracked tumbleweed on the head. Then let you out? Yeah. I was told to get here fast, so I came. Who was a cowman? I don't know. He said he had a lot of cattle. He was moving north. I saw him talking to Will's in the cafe. Will's warned about outlaws around here. Hold on. I wonder if he's the owner of the critters we just brought in. You mean it's... White Mustache and hair? That's the man. Hang it all. There's something minor funny about this. Why did he let you go free? Well, Pete, he didn't know his cattle was to be stole. No, it was a... What? What was he for guns? Mass man. No! You're the rest of the community. There's a load. A load of rangers in charge here.   They let you out so you'd show them where the hideout was. As a matter of fact, you were allowed to steal the cattle so you'd bring it here. I'm in, Lodge. Thought I was on your side. Didn't you, Bleak? You. The honorary two-horned polecat. Didn't you, Russers, think it was mighty odd there weren't no guards on those long horns? So that's it. I didn't knock old tumbleweed out, Bleak. In fact, he's right here with us. Right here and coming inside and speaking for myself. You thought I'd count out Ace Will's like every other snake in town. Thought I was too old and worthless to be a good sheriff. Well, I'm going to be the sheriff that cleaned out the crooks in Beacon City. You mean you talked about cows to Will's? No one need to have us get them? A lone ranger hoped he'd make a play for them. When Will's did so, that proved he was in with all you crooks. And while you crooks are busy stealing my friend's cattle, my deputy sheriff led the arrest of Ace Will's and his two sidekicks. Turner, we'll probably find a lot of cattle that's been stolen from other ranches. You will, you sure will. There's three times as many head here in this valley as I had. Well, we'll check the brands and notify the owner. Now, wait. Look here. Tumbleweed, maybe we can... Well, it's out of my hands, Bleak. That man with the mask is my deputy and he's in charge. You won't need me any more, Tumbleweed. I think Bob Turner and the honest men in town will back you from now on. We sure will. With Will's in jail and his gang smashed, the honest men will have a chance to say a few things. Now, wait. Great day. Don't you resign as deputy till we can find a jail big enough to hold this pack of crooks. You won't need a jail. No, of course you won't. Tumbleweed, we'll just give back the cattle and move away from here. That'll be all right, bonus. Well, I don't know how I can stop you. I got no jail big enough. What about a deputy? It's out of our hands, Tumbleweed. Huh? I hear Tunnel riding in. He'll have some men with him. Here's why you crooks get the surprise of your life. What do you mean? Those critters are mine that you sold. I reckon you sure will be sorry you stole that particular mess of long horns. How many men are stopping outside? Who all is there? A little Tunnel. Ah, they come. Soldiers outside. Good. Bleak, that cattle had been sold to the army. You stole the property of the United States and the troopers will take charge. Come on, soldiers. There you are. There's your prisoners. Wait a minute. It's the army prisoner for you. Won't they be surprised when he learns that he's a prisoner of Uncle Sam? Here's Tumbleweed. Here's the deputy badge. Hold on. Hold on. Stop that man. I gotta talk him into staying as my deputy sheriff. He's worth 100 regular deputies. Stop him. You won't need us now, Tumbleweed. Come on, Tuddu. Get him out of here. Get him out of here. Well done! Hooray! Before you have just heard is a copyrighted feature of the Lone Ranger Incorporated.


What we're going to talk about today is IP Anycast and how to implement it with DCOS or Mezos, everything that I'm showing you here should work equally well with open source like Mezos or DCOS or whatever. So my name is Bill Green, I'm a site reliability engineer at New Relic and what I'm showing you today is something that we are sort of experimenting with in our data center and I should probably just say from the outset that what I'm going to talk about is kind of a silver bullet, it has sort of a limited number of use cases but for those cases it works pretty well but I'm hoping that in covering these topics we'll go over some stuff that's useful anyway like you know how to do networking and what plugins are and things like that so I'm hoping that you know it's useful broadly speaking. So the first thing I guess we should talk about is so what is Anycast, it's one of those terms that you hear a lot and it's not real, maybe it's not real clear what it is. So from a layer two perspective like on an Ethernet LAN or whatever you have broadcast traffic and you have unicast traffic and you have multicast traffic and so the difference between them are sort of how the addresses are or how the frames are addressed to stations on the LAN so broadcast goes to everybody, unicast goes to one station and multicast goes to a group and the members of that group can elect themselves to be in the group. IP Anycast works at layer three and it's not so much a feature of something we'll go into that so what it is is it's where you have the same IP address that appears in multiple locations in the network and so you let the routing protocol decide which particular address is going to receive the traffic and so the way that this sort of works is that the routers you know keep have a routing table and they have the destinations in the routing table and normally the routers will pick the closest or the lowest metric or you know some sort of administrative thing to pick one route and put in the routing table and that's the active route so even though you may have multiple routes to a destination the router normally only picks one. With anycast it actually will make use of multiple routes and this sort of thing has been in use for a while and it's if you've ever used like you know Google's public resolver that the <IP_ADDRESS>.8 that's an anycast thing if you try to hit that IP address you'll get routed to like the closest <IP_ADDRESS> to you and it's not necessarily the same one every time so anycast like I said is an effect it's not a feature so if you go you know thumb and through your router manual looking for like how to turn on anycast you're not going to find it. The feature is called ECMP or Equal Cost MultiPath and all that usually is is it's a little knob in the config where you tell the router or the routing protocol be it BGP or OSPF or you know whatever routing protocol that you're using you tell the routing protocol I want you to use multiple routes so if you can see multiple paths to a particular destination I want you to use all of them and not just pick the best one but keep all of those in the routing table and it will do this if the routes are of equal cost and there's some other details in there like if you're running BGP typically it wants to keep the same destination AS but there's knobs to override that. This little knob for Equal Cost MultiPath will apply to BGP to OSPF both version 2 and 3 you can use it for MPLS if you're like advanced and it's usually as simple as just going into the configuration and saying how many routes you want to use and the router implementation will have limits on the number of equal cost routes that it will do. Usually it's on the order of like 1632 something like that so you would say like under your BGP config like I want you to use like you know four routes or whatever and that will tell it to enable this feature. So a contrived example of this let's say and I'm sorry for being US centric here my being an American I'm my geography is poor so I'm going to rely on what I know. So let's pretend like we have SFO, San Francisco, Chicago, New York and Dallas that you can imagine like a ring or something around the US. So if you're connected to San Francisco and you want to send something to something in New York you can go the northern route so San Francisco, Chicago, New York or you can go to the southern route San Francisco, Dallas, New York. So the SFO router will have you know in a contrived example two equal paths to that destination. So if you turn on equal cost multi-path in San Francisco it will it will put both of those routes in its routing table and so if it gets a packet from A to B it will basically load balance or load share on both of those paths.  So there's implications to doing this. One of them is that the ECMP will take care of the flows for you generally speaking. They will hash the destinations using some sort of hashing algorithm. Typically it's like they will take the source and destination IPs and ports and sort of hash those together in a group of buckets so that all of the packets in the same flow will go the same route and this avoids having you know packets from the same flow possibly being reordered if they're going in different paths. There's usually knobs in the router to effect how this hashing algorithm works. You can sometimes tell it to use you know source destination ports sometimes you can go as far down as like the Ethernet address if you wanted to. Usually on modern routers the hashing algorithm is a consistent algorithm so that if you were if one of these paths becomes unavailable you don't have to rebalance all of these hash you know the hash entries. You only lose the entries for the link that you lost and this has implications for things like you know if you're balancing long lived flows and things like that you don't want to disrupt everything if you lose one path and with that you usually don't have to. So we're going to talk a little bit now about routing in a data center. So we have this idea of anycast and so we have to talk about routing in a data center excuse me so we know how to sort of implement it. This is the usual this is the Cisco model that was popular for a long time and it still is in wide use and it has the idea of core distribution and access where you do this aggregation from like you know a lot of ports at top of rack and you sort of aggregate your traffic as you go north in the data center and so forth. This is not really a design that will scale. It does have some benefits like it's optimized to lower the per port cost at your top of rack that's really what it's designed to do. The drawbacks of this are pretty much everything else. It is really hard to scale this. It's expensive. It's usually there's usually a lot of layer two involved so you get these really large broadcast domains which are sometimes hard to troubleshoot. The alternative to doing it this way is with a so-called spine leaf topology. This is the more I'm going to say modern way to do it. This is becoming more popular in a data center. So what you do in this sort of design is you have your top of rack and you have a spine layer that's sort of your core and everything talks BGP so that each of your racks then become an autonomous system and they're kind of like their own little ISP or their own little network and so all of these things become little islands that exchange routing information with the spine as if the spine were the internet provider of these racks. At first glance it seems kind of crazy to do something like this but the advantages of doing this are that BGP is a very robust protocol. It's been in use forever. It's very mature. It's well understood and it's actually very simple to configure. It's generally as simple as just naming a route and saying hey I have this route. I want you to export this to everyone else and it will go do that. And so the topological features here are that you have usually a grouping of spines that follow a power of two. So you'll have like two, four, you know, eight, sixteen spines. And each top of rack is connected to each one of the spines. And equal cost multipath routing is used throughout a design like this and what ends up happening is that at the top of rack there if you can imagine that these are 10 gig links you have an aggregate bandwidth of 40 gigs out of the top of rack and it's load balanced among these different paths and if you lose one of those links the traffic will just shift over to the remaining three. So in the case of failure you just end up degrading your bandwidth a little bit. And you have like a numbers game where you can support a maximum number of racks based on the port density of your spine. Typically these are on the order of like 96 ports or something like that. So on a spine layer you'd support like 100 racks basically. And you would oversubscribe the bandwidth in your racks by like an order of maybe three to one or something where you might have 40 gigs aggregate out of your top of rack and in your rack you might have 30 servers connected at one gig each or something like that. So there's some oversubscription here so it requires you kind of be judicious about how you provision these things. So from a oh sorry. So the spine leaf topology so it's layer three everywhere you could do OSPF the more common that I've seen is BGP. Usually this is EBGP so it's external so every sort of thing here is an autonomous system among itself and it sort of like peers with everybody. The benefits are a massive scale out. You can sort of see here that everything every rack becomes one hop away. So it tends to equalize your latency so that every you don't have these crazy you know where you go all the way up to the top and back down to reach you know in a in a sort of more core access distribution model you don't have to go all the way up to that stack and back down. Each hop is basically just one bounce off the spine layer and so it's very low latency. And like I said with BGP it's simple it's robust it's mature. So what does this look like well you have like your top of racks you have your AS's and so you might have something like this where each rack is an AS. So you can actually bring this down to the host if you wanted to. You can run a BGP demon on the host and you can have the host in its own AS and then what you do is for everything that runs on the host you just simply announce a host route for that thing. Now your thing can be a container it can be a service it can be whatever and then in this way as your services come and go the routes are announced and withdrawn and so this sort of saves you from having to do like a load balancer to keep track of where everything is or do like complicated service discovery schemes to maintain you know to understand where your stuff is. But what does this look like as far as Docker goes because this is where the hard part comes in.  So you might spin up a container you would allocate a namespace to the container so the container gets its own copy of the network stack and then you build a virtual ethernet pair and you give one end to the container and then you give one end to the kernel and that's how you talk across the namespace as you do that kind of thing. You can either and this is just a point to point link doing this bridge topology thing where you give one side to the container and one side into the kernel that's no different from like a t1 link or something where you're like linking a remote office it's exactly the same thing except in this case you're not routing it you're bridging it so this is layer 2. So in the kernel you collect all of these virtual ethernet interfaces that go out to your various containers and you put them in a bridge. This is just like a layer 2 switch so it's like you're taking all of these remote offices or whatever and you're putting them on a switch so that you have a layer 2 and you you know send the frames everywhere and there's some kernel trickery that that comes into this but this is basically what you have and the most the important point is that all of the IP addresses that get allocated to the container come out of the same IP block so this is like you would Docker usually gives like 172 17 you know in this case it's like 0.1 and so every container gets an IP address on this network.  We would not be able to like go down to the to the individual layers of the Docker containers because there's only one interface here in the kernel and that's Docker 0 so you would have to either announce that you have all of the containers or none of them and so if one or two or three of your containers are not reachable you would black hole you would announce that you have them even though you don't. So what we want instead for this any cast scheme is we want what I just showed you except we want it routed we want to we want to still give every container its own namespace we want to make a pair of virtual ethernets we want to give one to the container and keep one but we want to route it we don't want to bridge it and the reason why we want to route it is because these become host routes so instead of having a bridge network we have these slash 32s which means they're just routes to a host and these routes go in the kernel table on the host and whenever you have routes in a kernel table you can then redistribute those via the BGP demon into the rest of your network and then as containers come and go these interfaces go down and come up or whatever and these host routes appear and disappear and thus they're announced and withdrawn from the network. So this is this is what we want to have and this allows us to we you can do IP per container this way so if you if you have some some IPAM scheme where you want to you know assign IP address to containers you can do this but in our case we want to assign the same IP address to a group of containers so we want to take you know marathon and we want to deploy you know in copies of an application with marathon we want each one of those instances to come up with this exact same IP address in different parts of the network and then we want this this IP address to be announced into the network so that all of the routers see multiple paths to the same destination or what they believe is the same destination but in reality it's multiple containers so that's that's what we're trying to achieve with this so the question is how do we do this and so that's sort of the heart of this so we have this idea of plugins with both docker and with mezzo so if you're not using the docker plugin or sorry if you're not using the docker container iser if you're instead using the mezzo's universal container iser this works in a very similar way and so on a on a debt on a mezzo's or DC us platform you have your choice of two different container isers in both of these cases you can write plugins for this container iser to customize the functionality that you want and they work a little bit differently but the end result is the same sort of so in the in the docker scheme they they have what's called the container network model and this is basically farmed out to a thing called libnetwork and so in this case you have a plug-in driver that runs in a container on the host and its job is to listen to API calls from docker to do things and these things are you know I need an address I want you to create a new network you know I have you know I want you to release the address so docker makes these various calls and your the custom plug-in actually does the work to do this and so this this container this docker plug-in is just an HTTP service it's no more complicated than that so you your plug-in is an HTTP service it runs as a container on the docker host and docker talks to it over HTTP and generally this is via a Unix socket and not a network socket because you don't generally want to make this available to the entire network and in the container network interface which is sort of like Kubernetes and the excuse me the universal containerizer that's what this uses and it's it's a similar scheme except it doesn't run as an HTTP service it's an executable and so the containerizer in this case just execs the executable and passes it arguments both via standard in and with environment variables but it's the same sort of thing it makes requests of the plug-in and says hey I have a thing I want you to create me a network I want you to assign an IP address etc the other difference here is there's really two parts to this there's the network driver and there's the IPAM driver and I'm gonna gloss over the IPAM driver but this is what actually allocates the network or allocates the IP address and so forth and it's sort of assumed if you have a custom IPAM driver that you have a larger IPAM implementation in your organization typically organizations like to centralize IP management for obvious reasons so like if you're gonna do the IPAM driver that's you kind of that's how you get it plugged into the rest of your your organization for these purposes I'm gonna assume that when you spin up a container you're just gonna tell it what IP address you want it to use and so it sort of bypasses the IPAM driver but you need to know that both of these plug-ins deal with that a little differently the Docker plug-in there's there's two separate drivers there's the network driver and the IPAM driver with CNI there's only the containerizer only talks to the network part and the network part is expected to talk to the IPAM driver on behalf of Docker but the functionality is the same it's just it works a little different in how the calls are made so ultimately what what we want to do is we want to you know have a routed network so this is where we start so usually the plug-in is given a container and the container is just a bare bones implementation of you know some sort of running process and it has its own namespace so what I'm trying to show you here is the dark blue is the namespace on the host the light blue is the namespace in the container and so the steps that you generally follow are you take this and he was you make a virtual ethernet pair and again this is what we're doing in the driver so the first step in the driver is you create the virtual ethernet pair and one end is going to go in the container the other end is going to go on the host you assign a dummy Mac address pair to assign to the virtual ethernet pair and you take the IP address and you assign the IP address to the interface in this case we're using the 169254 on the host side this is a link local address and it sort of gives us a way to route to basically do a route to an interface and I'll cover that in a minute but the host the host side of this is always going to be that exact same address and then what we do is we make routes on both of these so in the host side we take the host route of the container and we add that to the routing table so that the host now knows how to route to the container side in the container we need two routes we need a route that's a that's an interface route and we take the link local and we say okay this link local address can be reached by putting it on this link and then we make a second route that's a default route we point it at the link local address and this gets us out of the container and so this allows us to route out in order for this scheme to work you have to have proxy ARP enabled in on the host for at least the interfaces that are coming up and it's not usually an issue to do that but you do have to enable that and so at the end of all this this is what we have we have containers that are coming up with each container having an IP address the route to the container is contained in the host routing table you run a BGP demon on the host that redistributes these addresses out to the rest of the network and so what I'm showing you here is these 192 addresses these are the addresses that your host is numbered in so these are the this is the address space in your rack that you would have so each host is going to get like a 192 address but inside the host the individual containers are going to have a different IP address and it's going to be a host route and that the way that you tell the rest of the network that to route to your containers is you have to have the BGP peered with the rest of your network to tell it that you have these routes so if you have this scheme it will all just sort of work so sort of the use cases for this so why why would you ever want to do this there's a couple reasons why you would want to do this one of the reasons is for like UDP endpoints so like if you're running a syslog service if you know if you want to be able to you know just send your logs somewhere and you don't really you know if you have a few syslog services scattered around your network or whatever or if you have multiple data centers you might run a logging service in each data center you would want to be able to give you know an IP address into your host to be able to log to this stuff and so what you would do is you would configure your log service within any cast address so that all of your log collectors would have the exact same IP address and your host will use whichever one is topologically closer so that if you're in a data center let's say your host would tend to use the log aggregator that's in yours in that same data center but if somehow you lose that one you'll still hear the routes from the other data centers and so it will go to those instead it also works for things like you know like I mentioned DNS will do this for like it works really well for s-flow for net flow which are all UDP based services but you can also use it for TCP services there's not a restriction here so what you can do which is kind of interesting is you can deploy HA proxy and so the what you can sort of do is you can you can deploy your service like if you have a web service or whatever what you do is you deploy that and you use Minuteman if you're using DCOS to give it an IP address that's normally only reachable inside your cluster so that Minuteman assigns you know some sort of dummy address and you get a host-based VIP for your service so that you can run you know three copies of your of your HTTP service and you could put those behind a Minuteman VIP and if as long as you're in the cluster that totally works great but then the problem comes from like what if you're outside the cluster and so the usual solution for this is marathon LB or something similar where you run like an HA proxy so that relies on this sort of kooky scheme where you're like having to reload HA proxy like every time something changes and you know HA proxy was never designed to do this and they've gotten better about it but I I still am kind of wary of doing stuff like that so a better case I think is to just deploy an HA proxy container and assign the front-end like you know your server side of the HA proxy and any cast address and the backside becomes that named Minuteman VIP and it's static and that config never needs to change and so you deploy you know three copies of this with any cast and it gets deployed and so your your HA proxy becomes globally reachable from anywhere and if your containers move around Minuteman will take care of you know updating the you know the actual container instances but the HA proxy instance never needs to change you could do this with like if you're running you know traffic or you know any kind of load balancer that follows that kind of pattern you can deploy with any cast now the caveat to this comes from what happens when you lose you know when you lose one of those equal cost multi-path routes because the routers do have to rebalance those flows that you lose when you lose one of these routes and that causes the connections to be reset for just just the flows that we're using that one route the other thing is that if you deploy HA proxy using an IP any cast address you lose the ability to gracefully take your HA proxy instances out for maintenance so if you wanted to do something like this you would have to take the HA proxy instance down and it would reset all of the connections that are on there and they would be forced to use to reconnect to another HA proxy instance so as long as you've got applications that are willing to tolerate that it works well if you've got stuff that won't tolerate that or you've got or you know you're sending like large files or something via post or something like that were to reset these connections as problematic you don't want to do this because you'll run into issues so that's that's the sort of high level I glossed over a lot of the details but what I wanted to show you I guess I'm not gonna be able to do that so there's a I'll put on this slide deck when I upload it there's there's a repo from Adalia on GitHub that actually does the Docker plug-in that will enable the routed plug-in and it works out of the box you can just deploy it and as long as you use that as your Docker network driver it'll divvy out these these host routes so if you have a setup like this in your data center where you're running BGP or something else and if you're set up to where you can take your host and peer it with the rest of your network it's really easy to do this you just need to run a plug-in and the plug-in is called see CNM routed plug-in and you would just run this on your host and you would divvy out these host routes and suddenly your containers would be reachable from everywhere in the network and you wouldn't necessarily even have to use anycasts like I mentioned if you have like an IP per container scheme you could do this also and so it would make your IP addresses for your container reachable everywhere also so that's kind of the gist of the talk I wanted to open it up for questions because I realized I sort of went through this fast so if you know the word like if you've got four of the same IP address in one rack in one over here since the hosts are doing the announcement that the top of rack would get all four of those host routes and it would announce those so everywhere in the network you're just going to see five routes four of them happen to go to the top of rack are you thinking about aggregation so if they're on the same host in the rack that's probably not going to work because it will if you tell it to so that that's a thing like normally normally with a sort of garden variety networking you want to aggregate that stuff right you don't want all these little really specific routes leaked out everywhere because you end up with these gigantic routing tables so typically you would have something on the on the router that's like a you know aggregate supernet or something like that where the router will automatically take these more specific routes and aggregate them behind a single ad aggregation and as long as one or more of these more specifics are reachable this route stays announced and you don't have to do that you can you can specifically say hey if you've got anything in this net block just go ahead and announce the 32s into the network yeah so so the question was like if you if you're using it generally like IP per container and you're announcing all these slash 32s like what's the limit to that you don't want to have like 60,000 containers in a data center all announcing these host routes into your network it sort of understood that you're gonna have you know a small little bit smaller implementation than that but generally speaking you're gonna run out of IP addresses before you run out of you know this kind of thing I don't even if you're using private addresses if you have these just math like if you're Facebook you're gonna you know doing IP per container is gonna be very very difficult because even if you're using private addresses it's gonna get very difficult to divvy that stuff out one more question maybe two more good in the implementation I showed you yes now you could always get around that by you know I don't know you could probably use net filter trickery to get around that but generally like the kernel is going to make that decision so if the kernel has that route and it's local yeah it'll use whatever local thing I don't know of an instance where you want it to go out of the rack can you think of a like what's the use case oh yeah yeah right so like if you have like a bunch of stuff in that rack that needs to talk to that one thing yeah you would have that issue you'd have to put multiple things in that rack to sort of share the load in that case it it gets really hard to sort of granularly like do this kind of stuff with this scheme because it's not it's not really meant for load balancing in that sense it'll it'll like load share and not load balance if that's like a thing is there one more question like yeah so you it depends on how your IPAM is right so you can have your IPAM do your DNS for you so if you wanted to you can name these things and so you would have an IP address with a name and so in your service you know in your configs you would inject the host name and so the host name would just you would use DNS as your service discovery or you can use mesos DNS or console or whatever but but the IPAM that I sort of glossed over that's that's where all of that happens is like how you get that into the larger discovery thing generally and we're out of time thank you very much for attending I know it was kind of a very dense talk but I hope it was


It hasn't been an easy year but we're exceptionally proud of the journalism we've been able to do with your support so far. From our twice daily analysis of the coronavirus crisis to our original reporting of the labour leaks we've been working around the clock to provide the news and analysis you just won't find anywhere else. We're no longer a volunteer organisation being run out of a glorified shipping container. We're breaking stories and setting the agenda and giving the mainstream media a run for their money. 2021 is going to be Navara Media's most ambitious year yet. We want to do more podcasts, more original news gathering and allocate resources to do original investigative reporting. But to break new ground in 2021 we need to raise our monthly income and fast. That's what we're trying to raise an extra £5,000 in monthly donations by Christmas. You can help us do this in one of two ways. First, if you're not already a Navara Media supporter, now's the time to become one. Second, if you're already a supporter, we're asking you to increase your donation by a few pounds each month. However you want to support us, just head to navaramedia.com forward slash support. We cannot emphasise it enough. All the work we do depends on your donations. Who knows what's going to happen in 2021 but one thing is for certain, there's never been a better time to build a new media for a different politics and it can't happen without you.