Okay, good afternoon So we are here for the Octavia hands-on lab. My name is Gamma Nike burger and I'm joined by my colleagues from workspace Adam Harwell Franklin Neville Trevor Waterman Michael Johnson and the gentleman walking here is Stephen Balochor from blue box and IBM company And I will hand over to my colleague Adam who will do the first few slides Followed by Stephen then I will do the whole demo and thanks guys for showing up So yeah, as German said, I'm Adam Harwell. I'm from rack space and I'm an engineer on Octavia I'm gonna be helping you guys through the lab a little bit today. I'll be wandering around while German actually tells you what to do so you'll see me kind of in and out and as we go through just if you're having an issue just like raise your hand or Something and we have a few people who will be wandering through and trying to help answer questions and get you going So for the intro here see, how do I actually advance the slides? There we go. Okay, so we'll be doing a I'll be doing a quick pre-intro Then I'm gonna pass it off to Stephen in a minute to give an Architecture overview and dive a little bit deeper into the actual components that Octavia uses I don't know how many of you made it out to the Elbas talk yesterday We went over a lot of the same stuff But we're gonna dive a little bit deeper into that In his section and then we're gonna go directly into some operations So how to actually spin up a load balancer in a few configurations? We'll do just a little bit of looking at the configuration of Octavia and Get you used to the the actual low-level components and then we'll go into some troubleshooting steps and So first of all So we're gonna actually be giving you some VMs to do this in But if you have a DevStack instance set up already, this is what the screen looks like on Mac for VMware fusion Just as an example, but whatever you're using make sure that you have VTX enabled Or nested virtualization enabled. Otherwise, it'll be really slow And I know which one doesn't support this. Yeah, virtual box doesn't support this at all So it's gonna be a little slow, but you shouldn't fall behind because The VMs that we have don't have netches virtualization either. So this is gonna be it's gonna be a little interesting But We're gonna start off by I think Franklin here is handing out VM info. Did you already get everybody or are you just starting? Everybody has a credential Excellent. Okay. Yeah, so yeah, you're gonna log in as root on these These are all rack space 8 gig general-purpose instances so they're preset up with DevStack with with Octavia actually already installed so it's a little bit of a Difference from what we showed on the slides We opted to to do that just because the time is somewhat limited and we wanted to really get into the meat of things and not get Stuck on what's essentially just what DevStack does for you So for these VMs, you're gonna log in as root with the IP and the password that's on that little slip of paper And then immediately pretty much just su-stack and that'll put you as the stack user I have some in-it scripts set up there that like just by logging in it'll run some stuff one of the things it does is Set up sources DevStack's open RC admin admin, which makes you Essentially gives you an environment where you're set up as an as the main admin user for DevStack So all of our instructions are going to assume that you are Set up with your environment with admin admin Yeah, just for the purposes of this demo and just makes things a lot easier And the first thing that we're gonna have you do once you get in there is do a load balancer create Because as I said these aren't Nested virtualization enabled so this is gonna take a while So we're going to start off doing that and then I'll pass off to Steven and he'll do all of the walkthrough of the Components while that's happening So so that command that you see there's a new turn albaster load balancer create dash-dash name LB1 Private subnet do that immediately because it's gonna take a number of minutes for that to get going That's the longest step in this process and we're gonna have that run while I talk about the actual architecture of Octavia. Yeah, and actually we have it's there's gonna be some long commands that are kind of finicky So it might be easiest to copy paste them. So I'm gonna flip back here if you didn't already Go grab these slides to follow along. I recommend doing that We've got a cure QR code there that you can just like load up the page or it's it's a pretty easy short URL bit.do slash Octavia dash slides So hopefully everyone gets those open now and if you Don't get that and we move on and you need to get to it later I can walk around and help you load those up. So again, you're probably gonna want to have those open somewhere So give another couple seconds for that. Okay So yeah with that I will be walking around again So if you have any any questions at all just raise your hand and we'll get somebody over to you to help So passing it off to Steven for the architecture. Okay So as you all know Octavia is a open-source load balancer for specifically meant to work within the open stack environment it makes use of several other Open-stack services in order to accomplish the actual task of Delivering a load balancing service for the tenant. So I'm just gonna go through this slide and just in discuss What all the different components are of Octavia? Some of these will become And you know important if you're used to working in dev stack for example You can you'll be able to see some of the processes that I describe in your dev stack when in in the screen session They have there so starting with the upper left We have neutron and the load balancer version to user API handler in the Octavia driver so Within there that means that so the the neutron services Damon is actually running the the API for neutron Elbas version 2 So again if you're in your dev stack you you can see it in there It's also one of the things that that that in Adam's script that actually sets up these the VMs that you're working on You know that once you enable the features for for I think it's what Q. Elbas or something like that That's that that gets enabled and then when you enable Octavia it enables the Octavia driver. So Again all of the command line stuff that we're going to be doing is going to be using the neutron CLI because we're actually talking to Neutron Elbas version 2 and then on the back end once the CLI You know does its thing it sends it off to the Neutron API Neutron API then Sends it to the I guess the Elbas v2 sub component which then sends it to the Octavia driver Which then sends it to the Octavia API, which is the first thing After after you get out of Neutron you're talking to Octavia Octavia itself The the actual control components are made up of four different daemons The first is the API, which is obviously the front end that we meant You know eventually we're gonna actually have a CLI that can come directly to Octavia Right now you can as a user talk to the Octavia API But if you're actually looking at this lab to figure out how to run this in production We it is not recommended right now to expose the Octavia API because it doesn't have any authentication yet That's actually one of the things that will happen in Newton. So right now you want Typically you're gonna want the Octavia API to be running on the same box or you know within a controlled network environment So only the Neutron Elbas Neutron daemons can actually talk to the Octavia API at the present time anyway So the the Octavia API and the Neutron Elbas API are actually very similar in design that is by That is that's by design because They were sort of Designed in tandem Anyway, so looking at that the Octavia API again is its own daemon and It uses it you know It can communicate to and from the database for doing Sanity checking and so on and so forth when you actually issue a command to Octavia And to actually talk to the other components of Octavia it uses Oslo messaging so basically a rabbit queue So Octavia API takes in the the command from Neutron Sanitizes it make sure that we haven't got anything in there that that is you know improper input And then it sends it on on the queue for the Octavia worker the Octavia worker is a Daemon by itself again, and that handles basically receiving messages from the API and the orchestration of setting up M4 and Actually deploying services on the M4 altering how the services are configured based on CLI commands or commands from the client In this case the M4 is probably a if you're not familiar with Octavia at all then M4 is probably a term that is new to you M4 is actually a Latin term, and it means container Actually, it's a big jar that they used to ship wine in back in ancient days And the reason why we didn't just call it containers because well, it's it's a VM in some cases It can be a container in other cases and eventually we're thinking people will probably do this on bare metal devices as well But the idea is they're dynamically created and dynamically launched. There's actually as part of the the Dev stack script One of the things that it does in there is it actually creates a VM image using disk image builder and triple O based on Software that is within the Octavia repository So you get an M4 image and when you have when when when Octavia API and then the Octavia worker get a command To deploy a load balancer. They actually go out to Nova and have them launch an instance based on the compute nodes Which will be the container that that actually runs the load balancing services and when I say the load balancing services in this case Right now it's all HAProxy still so it's all M4 is run HAProxy and we have our own interface between the the Octavia control components and the M4 themselves to Do command and control with them so anyway and getting back here to this diagram So the the everything by the way, we've tried to abstract everything out So there's driver layers all over the place within Octavia so that you can actually replace certain components with other components Depending on the specifics of your environment. We figure that most people who are planning on using this Software pretty close to as is will probably need everything that's in there But we of course wanted to make sure that if you have Vendors with a proprietary technology that they want to apply that can do it better than the open source one Of course, we have driver layers in there so that you can specifically Replace the the open source component that we're distributing in the Octavia project with whatever driver you want So again Octavia worker It handles again commands from the API spins up shuts down whatever And reconfigures load balancing services depending on what the CLI says And then once if you're running normally the M4s will put off heartbeats every I think five seconds or something like that Which then the the health manager intercepts it gets those heartbeats And that's how it knows that the the M4a on the in the end is actually still live if one of the if those heartbeats start failing and Suddenly we go for a certain period which I think in Dev stack is 60 seconds But in production you might want to adjust that value, but if it goes for a certain period with no heartbeat We assume the M4a is dead then we take action to replace the failed M4a With a new one so that the server should be self-healing in that regard if you have you know a harbor failure within the cloud You know Octavia will will notice that and and re and you know spin up a new M4a and reconfigure the services you have on it there's also so there's there's a couple different modes in which Octavia can operate right now in the M4a specifically the first one we did was of course just stand alone So you have a single M4a that runs a single load balancing service But now within the Mitaka release we also have active standby Which means that for every single load balancing service that you deploy Every load balancer you deploy there's going to be two M4a one of them is going to be master one of them is going to be backup or active in standby and And if either of those fails then the the health manager will notice it and replace the one that that failed If the master fails and the the the standby will be promoted to the master Just between the two that the actual controller components here don't even affect that It's it's a pretty fast failover usually within a second or two and then When when the when the controller was sorry when the health manager spins up the new M4 It'll automatically make it the backup because we don't want to cause another blip in the front end client traffic That's going through it So you'll notice here also that everything that we that I've talked about is all just sort of control control stuff We're not talking about data plane stuff the data You know the the actual Requests that come into the front end of the load balancer and then go out the back end to the back end servers That all happens on the M4a itself So you can actually you can actually lose components of the control command and control layer and it won't interrupt the end services The client see that's obviously not ideal. You want these things running all the time But that is one of the features of the way Octavia works is it's it is all of the control is not in the data Data path so it's it's pretty nice that way Okay, the other thing is we also have a housekeeping manager Which manages if for example, you've just shut down load balancing services or deleted a load balancer and you have an M4 That's now empty the housekeeping manager will notice that and kill it and that you know frees up compute resources Michael I think there's a couple of the things that these do as well. Did you want to say anything about I'm missing or German kind so Okay, so the housekeeper is it us to more things we have a mode where we use spare pools So the so you can configure that the housekeeper manager will bring up and for us which are not configured yet When you do then the fail over it happens much much quicker because it just configure them Don't have to know about boot something or when you want to bring up new load balancers That happens much much quicker when you have a spare pool The other thing the housekeeper manager does is we talk to our enforce we arrest in the face which is with TLS protected and If you guys know TLS certificates expire and the housekeeping manager will automatically Reissue new certificates and replace expiring ones. So, yeah, so that's actually a good point I forgot to mention that yes the everything that comes from the control layer to the amphora is over a load balancer management network and it is It is encrypted with Bidirectional certificate based authentication and encryption. So whenever we launch an M4 for example, one of the things we shove into the Into the new M4 is a certificate that it's going to use to communicate with the With with the control here and the controller always uses a certificate And both ends do the the authentication to make sure that we're you know So that the controller knows that it's talking to the correct M4 and the M4 knows that it's getting Commands from a trusted source. So Oh, you want to go ahead and get into it, okay I was gonna mention two or three other things. It's real quick. Okay. So the other thing is again for TLS We we make use of the bargain bargain project for people to store their certificates So that we if we need to like launch a new M4 Well, the M4 is with the doing the TLS termination need to know the certificates But we let we let Barbara can manage the storage of those of course the compute drivers how we actually launched them for us We talked and over for that right now and then for any of the networking stuff by the way all these M4 right now they will hot plug into back-end server networks I'm sorry tenant networks in order to Connect to the back-end members and all of that has is happening through the the network driver Which talks to neutron so there you go and okay off to German now Okay, so let's talk about the operations you want to do and The first thing we wrote down is you need some demo VMs which will build a load balancers So, let's see if it all works here. I'm in Dev stack one on my local machine and usually we can just Copy things So, let's see the works So I did everything we said earlier Just do that Then it starts a VM. It's the thing about so Start the other VM Okay, you're good So, yeah, that's for the back-end servers, which will do the load balancing and do a quick novel list look what's going on See they're both active That's because There's zero switch is really quick So what happens a lot on my system is when I do those things that they don't work See so what you need to do then is I think it's sake group List And then you will see that a default one is that Copy that Let's set default. We need to put in this ID which is the annoying thing when you want is admin sometimes but Adam is reassured me that on his VM that won't happen See actually, there's a if you are having problems with any of these commands up here There's a secondary example. That's slightly just slightly different That you can compare against that I know should work on specifically on that VM And that's in dot profile in the in stacks home directory So if you cat that or vim that or whatever you can see I have a bunch of scripts set up to do basically the same steps That we're doing up here. So yeah, if you have any problems with these scripts check a look to check out those and Maybe you can see some hints, but yeah, still raise your hand and I can come help you to okay But be careful because it'll be it'll be ever so slightly different So if you start using those it might cause problems if you just try to switch back to these Okay, then let's Go to another novelists because I forgot the IP addresses of those things we want to start a web server there so to get in there just SSH Into the address and then you have to Say something like a cups win and It's in the slides. They can go in there and copy our Script which will be a load balancer which will be not below it will be a web thing exit the other one 10.04 Then we run it again Now we should have if you did everything right should have web servers running You see it says this and then we have that So now the thing is I'm probably very fast So did you guys get to that place or people still working on it? So who has this web server running? cool one So I've had a couple questions about the security groups because there's three defaults in that VM for now Just add that rule to all three of the defaults That's just to get us through this demo again the security groups are there to actually protect stuff But for the purposes demo, we just need to be able to talk between the VMs Obviously wouldn't do that in production, but again, we're just trying to get through this demo Well, you can figure out which one you're using if the nova sec group lease list command if you want to do that Okay So hopefully it's getting better if you guys want me to move on can do that So you should have done this new one command right at the beginning the load balancing rate. So I'm doing that here for me That will take a while So just starting that Okay, let's show you guys the command to check the status which is new one Which would also be a little bit further down albass load balancer minus list and Because I'm running VTX on my box. It's up. I don't know that should have happened for you too if you start right from the beginning Okay Does anyone still when they run that list command? Are you still showing it is pending create or People are still seeing active now Well, let me put a I guess show of hands for who's who sees it is active right now on theirs Oh good. Okay. Anyone still with pending create on theirs. Okay Okay, so when you do the next thing and it would be pending great. You will get an error Yeah, if it's pending create the next command will work. So what we now do is listen us That's basically so we created a load balancer Which is the IP address you are at and then you want to put listeners on it which Represent the ports want to make an HEP listener. So we putting something on port 80 listen I create load balancer I'll be one we're telling it on which load balancer and I miss saying telling the protocol which is HTTP and Most commonly that once on port 80. So we tell them that to name it listener one So that happens the next thing you need is a pool. So it's a listener a pool and then you have to add your members So let's do that pool create So basically the way to pull you can say which algorithm you like I like one drop in a lot You can do other ones too. You can need to tell it which listener it's going to listen to one But there's a way to do shared pools which we see later Then the protocol on the pool is HTTP and the name again Pool one, so we are there now. We need to add the members and This requires us to remember that our web servers there Let's see, and it was 10.003 and 10.004 in my example Takes a while 004 and You see there's also when you look at the bottom of this output as a weight So in case you would have a much more powerful member than the other you can change the weight So requests go to one member more than to the others So let's Curl that let's go back to our curl command. I think it should be on five or six. Let's see You do it again. It comes to four which is what round robbing does So so you see every time we curl it goes to three to four to three to four So this is how you set up a load balancer Let's see if people Got that far Okay, so who got there? Okay one two three four. Okay, so we should probably wait a little bit more before we move on Okay, so we probably Want to move on to the next thing so so we did a HEP load balancer But the world is going to be secure and so we want to set up a TLS load balancers the next one Or basically we set up a TLS listener under the same load balancer and The way things work in our system is we store all the certificates in another open stack project called Barbequen and so we need to figure out Where it is and we prepared every VM with With a certificate is already there. So when you do this Barbequen list Barbequen secret container list command it will tell you there's a Certificate and it says that the secrets Private keys and stuff and no consumers because nobody's using that right now. What's of interest is That's done in a way. I don't like it Is this container HVAP the container HVAP you see so this is the thing you need to pass into your listener. So Octavia or a neutral albastery to knows which certificate to use So and we put and we loaded them up for you because you're not a Barbequen workshop So you need to look there Okay Let's see next command Create another listener, which is supposed to be the TLS listener. You see here. I put in container HVAP This is this thing from above here So you have to Copy that Copy paste a lot in computers is about copy paste Did it right? That will come up to my HBS again. We need a How they're claiming we could use that pool Let's see That all works A copy paste our Okay, so so we create another pool for the HBS then we have to add the members again and Since I did it already I go back my command buffer pool two then we go And add the other one which is on three and then we can Go back to our curl This time So I will show you some things so when you do it that way then it says you can't verify it because we have a self signed Certificates we have to do curl minus K or install this stuff Then it says that but how do we know that it's actually a certificate? We did so we do curl minus CV And then we have it's done on localhost and You can look at a CN and stuff self-signed certificate And so on as you see we went to four so again the round robin That's how we That's how we can do as TLS one of those other great features and and what happens if Octavia is the TLS gets offloaded To the Octavia system you have seen we didn't have to change the web server still a web server Returning stuff on port 80, but now we can do TLS and so all the TLS workload is handled by Octavia Okay, let's see who has the TLS load balancer running our TLS listener here good two people Okay, let's So let's move on the next thing I wanted to show was L7 L7 is a feature we introduced with the mitaka release so it's brand new and It's basically that you can redirect things to different pools. And so we start with creating a new pool And we want to create this pool And you look at it We didn't give when you look at a pool create command last time we did every time we did a pool create command We specified a listener we didn't do it this time because it's a pool We want to use file 7 so we had to specify instead of a listener where to specify a load balancer See that here all the other commands with all the other pool create commands We had to specify a listener and now we specify load balancer, which means it's a Pools are now shareable in mitaka and you can Assign listeners later or you can assign look you can assign L7 rules So let's throw one of our members on this pool only one so we can see a difference Maybe use 10.003 and then we have to create L7 things So and the way L7 works is for those who haven't been in yesterday's talk We you can create one or an L7 policies and they can contain one or an L7 rules L7 policies if you have more than one they are connected with or so it's the first one or the second one or the third one Whereas when you look at the rules, they have to be and so all the rules have to be in been be true that a policy gets executed so when we So here we have a very simple policy and you can And can do much more sophisticated things, but here basically we are creating a policy on listener one calling it L7 policy one and Two important things are the action Which we say is redirect to pool so when it matches one of the rules in that policy you will redirect it to the pool and We specify as the pool it should redirect to the pool free We just created and we are attaching it this L7 policy to the first listener and then we want to put a rule on it as I said the rules are and Here we want to do one a greater new rule I want to have a couple of things can do can do path cookie and so on and we want to put it on the path Want to say it starts with and when it starts with slash lab the URL then we want to redirect it to this other pool So that's done now when we do our curl from before not the HTTPS HTTP curl and We attach lab to it to the path Then we see it's going always to the to the web server on 1003 before that round robin so when I go without lab with three four three Four, but when we add lab to it will always be 1003 since we only specified one web server there and this shows that L7 works the way we want it So everybody has some load balancing stuff running by now Okay, good. So so this was the stuff what maybe end users might do to set up load balancers and listeners now we're getting more to the things an operator might might do and Let's get some information about the one and four we have So so as we said earlier the m4 are normal Nova VMs and so we can do a As admin of course you can do a Nova list and then you see here my m4 and Minor VM which houses the m4 it's active It's running and what's the couple of things so it has this private thing which we then mapped to the VIP and What's more important for us right now is it's as on our management network so all the m4 get booted in our management network and that's how we can talk to them and We will do that now so so go so there's an so and we have an agent running on that which you can talk to and Somebody standing in front of my projector Adam Okay, so so basically you can so basically we keep the certificates at least on death stack into it is the Octavia search client Pam and Okay post me So really quick I know there's a lot of people having issues with the security group step Who has still not gotten past getting the security group set up to be able to SSH into your member nodes Is there anyone still? Not there yet Okay, cool. That's that's good if you Still need need help with that. Just raise your hand and let me know I just wanted to make sure we could move on because I think we're getting a little bit of a head So we'll talk a little bit more about our agent interface So we put in a bunch of commands like info which will so if you're an operator You want to know what what version of h.a. Box? I'm running you can check that out which host name which API version Also have other commands like details, which is pretty neat So this is talking straight to our I'm for and here we get all kinds of information So we get something about a CPU the disk space we can monitor how many h.a. Boxes are running What's the load is on this box? How much memory what networks we have and And we didn't implement the packages thing. So in which topology you're running like single can also If you if you want you can also see what's going on there. How many Listeners you have So this is all stuff you might use to troubleshoot if you don't want to or you can't look into your and for Okay, so here you see all the listeners we have we have like The hdp listener and the hdps listener so to listeners you can even get more info by Putting in the idea of a listener So hopefully that works then it tells you about members pools Status up So here you see no check because I was Because we didn't define health checks on those health monitors on those Members, but if there would be health monitors you would see the status of those checks So this is a very good way if you need to look into things and Figure stuff out without using the nova API So the other way to learn more about the system is by just as is aging into this I'm for Let's find our IP again Okay, so this is a little bit Tricky so so we So you can configure it that it installs an SSH key on each and force again SSH and we do it on DevStack for Debug and everything But if you're running production, you might not do it that way Depending on what you want to do but anyway, so we stored the the private key to log in in Octavia dot SSH Octavia SSH key, then you can just log into an on for and Once you are in Can check on a bunch of stuff? So we have the agent which we talked to so so far. Let's see if it's how it's doing and Then you see oh no for agent is running. So we put it in as a system service You can also check on HAPoxy, but then you would need the listener ID. So so the way we Start them up is per listener. We start an HAPoxy and You need to know the listener ID Which was here somewhere Let's pick one like this one See listener ID So this is how we name those and then you see there's one running for this listener So so if somebody comes to you and says hey My port 80 thing doesn't work then you can go in there and see if the HAPoxy runs for that Okay So this is what we can do inside DVM bunch of Learning more about stuff So let's see are we at a point where you should continue or let this stuff settle Hey German I've had a couple of questions about People seem to be confused as to what the difference is between Elbas v2 and Octavia and where things are going in the future Do you want me to talk about that for a bit here German? Okay, so I'm sorry if I'm kind of interrupting the flow of what you're doing but so Elbas v2 the API is not going away So if you're developing a cloud software or software which interfaces with OpenStack You can consider Elbas v2 to be set in stone any changes to that API are only going to be additive at this point. Oh Thank you perfect So we actually just had a discussion among the groups who maintain Elbas v2 and the Octavia group Which by the way, there's almost an exact one-to-one correlation between the cores between the two groups They're they're very very similar You know the same people are developing the same software on both sides In the future you may see that a lot of the Elbas v2 functionality that is presently in that project in a separate repo may end up in the Octavia repo at the same time the the The Neutron Elbas v2 API endpoint is not going to go away and it's not going to change So we you know maybe in a year or so we might end up seeing it be a straight pass through to the Octavia API But at this point it should be safe for you guys to develop software which talks to Neutron Elbas v2 It's not going to change. Yeah, and and on that note if you're if you're using Elbas v1 Now is the time to get off of it. It's it's deprecated. It's going away Yeah, Elbas v1 is it's going away, but basically the difference as I said as Elbas v2 up in the corner and And you can and that's basically the API interface with all the functionality I showed and if you go out and buy a hardware load balancer, which is compatible with Elbas v2 would work the same way Octavia is basically a software load balancer you get for free from us and Blacks into Elbas v2 So that's basically a difference. So so so if you're buying stuff You get Elbas v2 by your stuff and what you see down there where it starts Octavia API all this stuff would be gone It would be whatever a 5 a 10 Yep, and so on and and we are basically Octavia's basically It's similar to a to a hardware load balancer or software load balancer Whatever. Yeah, those people are selling those days And if you're looking for where this code all lives The the the stuff there under the little neutron box. That's in the neutron dash Elbas project Everything else is in the Octavia project. Yeah, so the horizon integration right now It does work with Elbas v2. There was actually a demo of that yesterday. So You know in everything we've done except for the l7 stuff And I believe some of the TLS stuff you can't do in the gooey yet We're hoping that'll end in Newton and that right now talks to the neutron Elbas API and Elbas v2 API and it will continue to do so for the foreseeable future So yes, the horizon gooey isn't going away either Yeah, so as of the heat integration is up on Elbas v2 So in case you are wondering that's all the other software horizon heat Ansible Albas puppet Albas They're developing it's all against Albas v2 So it would work with any load bands are just compatible with Albas v2 Just just think of Octavia as like a load balancer appliance like any other vendor out there It just happens to be an open-source vendor second So the So Octavia is the reference implementation for Elbas v2 at this point So that's sort of the thing if you're not going to use Octavia Then you're probably going to end up using a commercial appliance to back you do your load balancing instead of Octavia at this point The the namespace driver does exist, but I believe it's I think it's deprecated. That's okay. Did you yeah? So the way that Octavia like in the Liberty release Where you could do all of the Elbas v2 features Using the model where it ran HEProxy right there on the network node That is not the recommended way to deploy And specifically it's because if you do anyone doing TLS termination That's actually very CPU intensive and it's very easy to overload your networking box with doing a whole bunch of SSL sessions This is why we do all of this stuff the M4 is all live on compute because computer design to do CPU intensive stuff Yeah, so Yeah, so Octavia is the reference implementation for downloaded def stag I want to do a ref and the reason we have a reference limitation is to don't let you add new API functionality like L7 Without having a reference implementation for that. So so that's why we need a reference implementation We decided to make a reference implementation which doesn't suck. And so it came back to Octavia Because before you could do reference and do toy clouds when you went somewhere then Didn't scale wasn't a lot of things Okay, one more question What other VM? Yeah, yeah, you can use albeas v2 at the same time as Yeah, albeas v2 is the API so you can use it if Octavia there's a You can even have Octavia alongside a net scale or whatever you can use as many load balancers as you can afford No, the M4 instances get created against the Octavia tenant Which which is an admin tenant and we usually in our installations we make unlimited quotas It gets created as a nova VM Whatever whatever and so so basically you can you can specify your flavor Okay, and whatever you put behind the flavor. That's where it gets created So if you think you have very good hardware for networking just make it a flavor and specify that and then all the VMs end up there We use no definitely yet. This is a nova flavor. So so so we do custom Yeah, we and also as we do we we set up a custom Octavia flavor So because you know when you don't make it custom and hide it and end users might find it and change it and break it Yeah, it's a nova flavor. You can say how much memory you want how much disk space how much CPU, you know Yeah, yeah, that's because we got a nova yeah, that's yeah, yeah, we didn't want to invent another scheduler So we think so we believe nova is smart enough to put things where they belong Thanks. Yeah, and you did talk about the anti affinity already So so so if you do active passive If you disobeyed a couple of topologies, one of them is active passive then if it active passive you might want to avoid putting There's two VMs on the same server or even the same rack And so we so then you can use and I affinity rules which we also define in Novak Sort of so you can tell them I want to be on different racks and then you slot it in can configure that in our end Then we create them That then then we tell Nova this and I finish the rule and then they put them on different racks different servers Different data centers. However, you configure that stuff and I affinity. That's a nova Yeah, Nova has an affinity rules and we have a field where I can put in a reference to this and I affinity Think they call and I affinity group or and I affinity group or something Okay So there was a couple more things I wanted to show Yeah, 20 minutes, right Okay, where did we get and the operations thing information about you for we looked into an m4 Failover Cool. So this is something we probably doesn't work very well on your VMs just warning you So as I said, there are a bunch of top apologies we support and And and and basically we have the failover means that so we have as Stephen said we have a health monitor which monitors the health of an m4. So if an m4 dies And we don't get a heartbeat back Then we go and and basically fail it over so then we we provision a new Nova VM with the same configuration and then we Delete the old one and and move the IP over and and now if you We said m4 are done are driven by images. So you have grants images Which which we use to build them for now it happens a lot There's a security problem or something and you have to make a new image So then you've then you kind of patch your image use the latest Ubuntu or whatever Which doesn't have a security problem, but you're still running VMs with the security problem And so you need to then kind of replace those VMs those enforce with the patched ones and this is what you would do by forcing a failover and I want to show that so how that works so we go and Probably start with a novel list First of all we get out of the m4 Nova list so this is something to watch Yeah, okay, so here we have them for and you see it's on 192 whatever o4 So we want to figure out which port it's using For that can do this and then what we want to do is basically So it's sending health information on this port to the control plane Then we want to shut down this port Which then forces the system to not receive any Think that's the ID when I see it right We do that so it goes down So we could run the curl and then we want to do a novel list again and It's still the one it is Yeah, it takes a minute or something the minute notice it it will Yeah, really takes a minute so it gives us some time in div stack the time out of set to 60 seconds in production You might want to adjust that It depends if you want active passive you could leave it at a minute That's true in active passive the passive node will notice usually within a second if the master goes down Or the active node goes down and then it'll take over But this is just a single standalone topology that we're doing which obviously isn't highly available No, it's still highly available. Well, yes fail over Just don't get nine point nine nine nine nine nine nine ninety five or whatever percent Actually much larger than it can be so the dev stack fail over time I think five seconds or more it's it's longer just as the default but you can configure that Okay, so for people paying attention now we have See that the IP changed and also probably the ID changed. So so now we are running a new M4 So so so it went out provisioning new M4 and replace the the one we were running before And it still is on the same Address in the tenant network. So the tenant shouldn't have noticed anything Was the same for him well except for you know a little bit of an outage right there Yeah short outage Okay, so so we talked a lot about topology and The next thing we wanted to do is show you guys the configuration file So we keep them on dev stack in Octavia Octavia conf and They are hard to read because they're blue Oh and set BG equals dark You're in you're in VI right? Yeah, Colin Set BG equals dark said space BG equals dark BG BG BG is in a background See how often I use those things Okay, good. That looks much better Very nice. Thanks Okay, so so so we'll go a few things Because I'm trying to cut a little bit on time to have more time for questions So so at the very beginning if you want to debug there's two settings verbose and debug They're both set to false. So your log files are not as big as they could be but if you want to debug problems Yeah, we So so if you do develop we often set debug up to true So we see the debug logs. I mean you said verbose to true You get the info logs and when you don't do anything then you get a warning and hire then database health manager Bunch bunch of stuff somewhere Talks about topology Okay, so here see topology options If you like something like active passive or active standby Then you have to replace this single here with active standby Then of course, then you have to restart a lot of stuff So if to reconnect The easiest way to restart things on DevStack is to reconnect to the screen There's a lot of stuff and you see all our services here on DevStack for some reason We stopped running the housekeeping manager, but so we have OAP I so we stopped that restart controller worker Stop restart and the health manager stop restart We detach and then we could do active passive. So we need to create a second load balancer So I will just do it very Great, I'll go crazy So we are starting a new one pending create and now it has to start To I'm for so it takes a bit bit longer should be rather quick on my box since I'm running it as BTX so both active Can look at Nova what we ended up with And you see we have now three VMs And Think the top ones are the new ones six and seven and they built this active passive pair 12 minutes left Well, so anyways, so we probably should You probably need a listener there quick listen I create probably not Let's do a Let's go crazy to an TLS listener So I guess I think I didn't change the name. So you can as you use IDs for things to Remember Put some members behind it That's not a good sign That's because I need to HBS Okay, that works. Now we want to find out who's the master is I want to use Easiest way to quickly go into the database and check for that. So there's a better command in the example But you see we have our stand alone load balancer Before we have a backup of a master and now we see the master is on 192 168 0.6 and then if we go and what we should make that More impressive Was it watch? Hmm. I did something wrong Okay, there we go Hopefully it does what I wanted to do. Oh, no, it doesn't gosh You do HTTPS? Okay Connection refused is annoying Whatever Does it do that maybe See if you can get a list That's seven. That's why gosh So now we are in business then we said which one did we say was the master was the Oh six that's this one now we do a nova delete Which is the fastest one to get rid of things Then when we do a nova list then we would Then we should see That we only have two and four left And we see on the other the other screen That it's still working. It was a small blip, but other than that that was what it took to do the Active passive failover now Since we did a nova delete the system won't automatically build a new M4 because it thinks when you do a nova delete That's what actually want get rid of things But but yeah, that's active passive Then I think we have seven more minutes. There's a bunch of troubleshooting stuff. We talked about a log level I showed you how to restart things So there are log files probably should there an abstract locks can should probably show you that too and All our op files our log files start with oh something and so there's oh API, which is the API log files OCW is there The worker log files and HM is the health manager log files You can look at those when you want to troubleshoot and see what's going on then we have Then you can look at the log files on M4. Oh, I killed that one. That's why Five so on the M4 we also have log files and we store them in a place called upstart This is where we keep the H and for agent log files. So so this is There can learn more about how our agent works What it's doing and look there it talks about all the requests it gets and what's it has to do So if we need to debug something that's a good place and I'll still want to show kba database showed it briefly and How to contribute actually wanted to have Michael talk about that but he's busy So I will talk about it briefly So we are OpenStack project in under the neutron in inside a neutron stadium and We have weekly meetings at 20 UTC in OpenStack meeting alt We have our own IRC channel Which which is OpenStack albass and there's almost always somebody there But not always somebody who can answer questions. There's always somebody they are asking questions And then we have a bunch of websites with our wiki Octavia We have our own Octavia IO where we keep our proposals and documentation Have a launchpad area and We have a github area where we put all our code and We are always happy for people to join us and help us write better code test it and whatever So if you guys are excited about Octavia want to help Wednesday or other days in our channel Okay So we have like four minutes left for questions any more questions stuff. I didn't answer Then I guess we are I Guess I give you guys three minutes back. You can still keep playing. Thanks guys for coming and So also blast thanks