Hello everyone. So now we have a talk on analyzing COVID vaccine popularity in Europe using fast API Postgres and Azure. Our talk will be by Ravya Imad. She's a program manager at Microsoft with a focus on building great cloud products for Python web developers in Azure. She has a degree in computer science and astrophysics from Harvard and is currently studying data analytics at Georgia Tech. Hello Ravya. Hi Yasmin, nice to meet you. Thank you very much. We're very happy to have you today. Wonderful. So yeah, is it time to just go ahead and get started now? Yes, definitely. Awesome. Awesome. Well, hello everyone in Europe and elsewhere who are joining me today. I'm so honored to have the privilege of your audience and I have a lot of exciting things for you today. So first off, a huge shout out to the Python community from Microsoft. We'd love working with you and building great tools for you. And thank you, Europe Python, for having us here. As Yasmin mentioned, the title of the talk that I'm going to be speaking to today is really analyzing COVID vaccine distribution by type in Europe. I'm going to be using a few different technologies like Fast API, Python Fire, Azure Web Apps and Azure Database for Postgres SQL, a flexible server edition for this. Some of these technologies are new and I'm going to be giving you a little bit of a tasting on all the different things that these technologies bring. But before I do that, I want to give you just a little bit of context about me. My name is Rabia as Yasmin mentioned and I'm a program manager at Microsoft and I work on the Python team. And so I am really a good contact for you if you want to learn about anything that's going on in the Python world at Microsoft. I've left my Twitter and my LinkedIn in the slide deck. You can also download this separately. And yeah, don't hesitate to reach out, message me directly or otherwise and yeah, we'll stay in touch even after the conference. So with that said, let's go ahead and get started. And I'm just because we have a lot of ground to cover today in 30 minutes, I'm going to give you a brief tasting of what we're going to be building today. And really what I'm going to do is build the architecture that's shown in this diagram. So I'm going to transfer COVID data to Azure Postgres. That's the database that's shown on the right hand side. And I'm going to connect it to Azure App Service, which is basically going to host a website. Then I'm going to use Jupyter to talk to that website and then visualize the data in a meaningful way so that I can get insights. And then of course, I can use my computer to go to Jupyter and actually see the results and share them with you. So ultimately, that's the goal of this talk today. And yeah, let's see what we need to go ahead and get started. There are a few things that I already installed on my machine before this talk. So I have Python 3 installed. I have Visual Studio Code installed. I also have an Azure free account. You can go ahead and get some free credit so that you can also try this exercise yourself and learn as you go along. And then I also have the Azure CLI installed. This is going to be important as I run certain Azure specific commands in my terminal. I've also already cloned the code that I'm going to be using from GitHub repository. And I have the dependencies installed as well as the virtual environment. And don't worry so much about following and retaining all of the information that I'm going to present to you in your head. Detailed instructions on what I'm going to be doing are going to be in the clone repository in the readme. The slides are going to be available for you. So just come along for the journey and have some fun and see what wonderful things Python can do. So with that, let's talk about data. Let's talk about what we need to do to understand data to formulate some interesting questions and filter some relevant components. So when I was thinking about the COVID vaccines and Europe, I wanted to go and get some data. And I did that by going to the website of the European Center for Disease Control that's shown over here. And I downloaded the data from the ECBC to my local repository. And there were a lot of columns in this data set. It was very rich. But there were really three that really piqued my interest. The first one was the number of doses that a country received from a given manufacturer, the name of the country and the type of vaccine, the name of the manufacturer. And if you look at the raw data, you know, it has a lot of abbreviations like the country name or the vaccine manufacturer names. They're not really spelled out completely. So I did a little bit of digging and I figured out what those meant. And so I made dictionaries in Python to kind of use them so that, you know, when I share my results with you, they don't have all these acronyms. So over here, I have the vaccine types. And you can see that this J-A-N-S-S, this is the Johnson & Johnson vaccine. And this A-T reporting country is, in fact, Austria. So once I got this data, I really got curious about, you know, what are the most commonly used vaccines for every country in Europe? And what are the least popular or least common ones for the country? And really, what is the difference between how common they are? Are they more evenly distributed? Are they not? Does it really change by country? So there are lots of different questions that I wanted to answer. And so with that, I had a few things to go right off of. And so, yeah, time to go and analyze the data and get the answers to these questions. So the very first thing that I needed to do was actually set up the database and the local connection variables so that I could actually do something with this database later on, such as actually transferring the data maybe and maybe retrieving it.  And you can see that my virtual environment is activated and I've installed all the dependencies. I mentioned this already in the prerequisite. And instead of actually writing down all the commands myself in a live demo, I have a really cool script so that I can just write step zero and it's going to tell me what it's going to execute. And then that way I can actually spend less time making typing mistakes and more time actually explaining things to you in a clear and concise manner. So step zero is something called AZ login. You can see this over here. And what it does is I'm trying to actually log into Azure from my terminal so that I can access the resources that I have there. And you can see that it's asking me to sign into my account. And after a brief period of time, it says, oh, yeah, you've logged into the account. You can close your window or just go back to your step. So great. We've already done the login. And now we have to do something else, which is run this command, which is Azure config param persist on. Now, you might not know what this means, but essentially what happens is that whenever you're deploying resources in Azure, there are a lot of different parameters that you need to set up. And you need to mention them in your commands. And sometimes that means that the commands can appear to be rather long and repetitive. So if you have AZ config param persist on, you can use shorter commands and it will remember some common things like the names you used and things like that so that you don't have to be typing them all the time. So once that is done, I'm ready to actually go ahead and talk to Azure. And the first thing I'm going to do is I'm actually going to go ahead and create a database in Azure. And that's going to be here in step two. And the command that I'm going to use is this one. It's AZ Postgres Flexible Server Create. Now, what this is going to do, like this part of it is it's going to tell Azure that I want a database that's a Postgres database and I want it to be of the flexible server type. Now, that's a Microsoft Azure specific type. And what it is, is it has a free offering included in it. And so that's why I want to use it because I want you to have the best, but the freest possible resources for you for your experimentation. And then you can also see that I've opened the access to this database to everyone using dash dash public access all. Now, generally, whenever you're procuring a database, you want to have a restricted access, but I didn't want to delve into firewall rules just yet in this lightning talk. So I'm just going to open this for now, but please look at the docs and see how you can configure the firewall in a more restricted manner. And then finally, there are some other parameters here, and those are really just needed to make sure that I can use the free tier because Microsoft provides a 12 month free version of Postgres Flexible Server if you procure it using these parameters and I certainly want to do that. So that's why I have these there. Now, you can see that as soon as I executed the command, you know, lots of things were being printed and this command is running. Now, the run for this command actually takes a few minutes in line with the run it would take on any other cloud. But I don't want to spend time on seeing this run completely for this demo because, you know, I want to be precise about it. So what I did before the demo is that I actually went ahead and I ran this in advance and I'm going to switch the tab now and you can see what happens when I complete finish completely finished running this test. So what happened over here is that once it's done running, Azure gives you an output, which is which is this green output. And this has some very important information that you want to keep. And so what you can do is you can copy this and put it in a safe, non checked in location on your computer, just for reference later on. So you can see that there is a database host name, a database password, a user name, and then also over here, it will say that it's creating a Postgres flexible server DB. And that is really the name of your database. So let me also save that here and then go back to my very first terminal, which is still going to be running. And I'm just going to go ahead and kill the process right now. Since I've already shown you what the results are. So with that, what we now need to do is we have a database created in the cloud, but I need to actually go ahead and connect to it so that I can maybe talk to it from my computer. And the way to do that is going to be shown to you in the next step. And I'll walk through what that is. And it's by executing these four commands. And what I've done here is that I'm exporting something called the server name, the admin username, the admin password, and the DB name. And these were all things that I got from. This is the database name. This is the host name. This is the password. This is the username. And you might notice that the ones that I have here are a little bit different. Those are really for security purposes. I wanted to do a demo with different passwords than the one that I was going to show in the slides. But anyway, with this done, I now have some variables that have been exported. But again, these variables are just in my local terminal. I actually need to do something with them. And what I need to do is I need to create a connection string. And a connection string is basically the combination of these things, the server name, the database name, the username, the password, and stuff like that. So that can be done by executing this line of code. And once this line has been executed, you can run echo on the connection string variable and make sure that it actually looks proper, which it does. So I'm going to go ahead and copy this connection string. And I'm going to paste this again in my little notepad that I have not checked in, just for now. And it's going to become very relevant in a few seconds. Now, why am I naming my variables, connection string and all of those things? That's because of the design of my code. And I'm going to get to that in a brief moment. But first, before I do that, let me do a quick recap of what we just learned. And what we learned was that we need to actually activate the virtual environment and install the dependencies, log into Azure and persist the parameters. Then what we need to do is create the Azure database for Postgres SQL flexible server. We had this command. And then there were all these credentials that we got from that command. And then we use the export name, sorry, we use the export command to export these variables and finally create a database connection string. And really, the role of this connection string is to connect me to the Azure database for Postgres SQL flexible server here, which is at the very bottom of the slide. And I'll show you how that can be done in a brief moment. So next, what I'm going to teach you is how to load the data that you downloaded from the ECDC website and actually put it in your cloud database and then retrieve it to verify that you've done the right thing. And to do this, I'm going to actually be calling a function called load data. And this function is defined in my data work dot py file, you can see that it's over here. And it's really using a file called COVID data dot CSV. This is a giant CSV file that I got from the ECDC website and it has many columns. But as I mentioned earlier, we are only going to care about a few different columns. And so what load data really does is that it creates a table called vaccine data, this is the most relevant part of it. And what this table does is it takes the number of doses that you received, the reporting country information, as well as the vaccine type information, and it puts it in a data table called vaccine data. For references also keeps the raw data in a separate table just as a best practice, but you don't really need to have that. And then finally, what it does is it tells you that the data was loaded successfully. Another very important thing that I'm going to be doing is actually verifying that the data was loaded correctly. And that's by calling something called get all data and using it to literally print out the three columns from the vaccine data. Now, these two functions are really important, but I want to be able to call them from the CLI. And this is really where Python fire comes in. And what Python fire does is that it lets you call these functions through the CLI using commands like this. So if I write load data from the CLI, it's actually going to call load data.  So that's the way of calling all your functions very quickly and in an elegant manner directly from the CLI. And we're going to use Python fire for that. So let's let's do that. Let's go on to the next step, which is step six. Oh, and one more thing. Remember how we talked a lot about connection string and connection string is really important because in order to load data, you have to connect to the database and in my code, it actually gets an environment variable called connection string and then use that along with Psycho PG2, which is a Python driver for Postgres to actually connect to the database. So that is the importance of having that connection string variable. So now that that's done, let's execute step six, which is simply this command Python three data work dot py load data. And what this is doing is actually going ahead connecting to that database and putting all the data both third and raw into the database. And it says that the data was loaded successfully. So that's great. And now I want to make sure that this is that the data actually looks like what I think it should look like, which is three columns. And so I'm going to call step seven, which is going to get all data. Hang on a second. You can see this is the command here Python three data work dot py get all data. And you can see that Yep, it's a lot of tuples that are three in number. So with that done, I'm actually able to connect to Azure from my local machine using a Psycho PG2, as well as all of these connection strings that I had. So let's do a quick recap. What we did here was we loaded the data, we refined it, and then we retrieved it using some code that we use Python fire to call using load data, you could pass in a specific CSV file, or you could just use the default name, either one works given the way this code is written. And then again, we did data verifications using the get all data flag. So now we have our database piece really complete. Now maybe let's do some interesting manipulations with the data. And that is really the next section of this talk, which is gathering insights and serving that slowly with the backend AI that built using fast API. And the way to do that is, let's go on to a quick demonstration of the things that you can do. So I've talked about how, you know, we got lots of different types of data. But really what I'm interested in was the vaccine percentages. And so I have a function here that actually goes ahead and computes the percentage share of every vaccine manufacturer for every country that's in the data set. And I'm not going to go into like how the function works. It's generally just simple arithmetic that it's doing. It's counting the numbers, and it's, it's spitting out a percentage. And you can actually verify that it can do that by again invoking, let's see, oops, we already did step seven, we can do vaccine counts. And then what I just did was simply call the function that would give me a vaccine count for every country. And I piped it using jq so that it's pretty printed. And so for example, you can see for Luxembourg, these are the counts of the doses that have been received per unit for Luxembourg and for all the other countries. So this is really cool data. But you know, maybe the counts are actually a bit overwhelming to process. So let's see if I can get some, sorry, some some percentages. And the way to do that is to go ahead and run this the same command. So the command that we just did. But instead of get country vaccine counts, you can actually even get country vaccine percentages. And so I'm just going to go ahead and do that here. And now I can see, for example, for Austria, AstraZeneca has 24% share, Pfizer is at 63, G&J at 2, Moderna at 9.76. So this is much more useful data. But now what I want to do is, you know, it's all well and good that it's on my machine. But you know, maybe and like a website that can build actually has the data so that I can analyze it and others can analyze it. And this is really where fast API comes in and fast API is a wonderful framework for building Python back ends. And the best part about it is, well, it's fast because it's asynchronous, it supports async code natively. And it also allows you to force a certain response type in your API responses. And so that can also really help you in making sure that you get the right type of data. And fast API is just like, wonderful to work with, I will highly recommend, particularly people who are new to building API is to certainly check it out. And so I have something here, which is I'm initializing fast API, and I have an app, and then there is a backslash pair, which is the root. And root is supposed to give you get country vaccine percentages. I also have something called data, which gives you the same. Then I also have an endpoint called most common, which gives me the top vaccine by the country, as well as the least common, which gives me the least common vaccine for the country. So with that done, let's figure out a way to actually get this up and running. And to do that, I accidentally ran this command quickly before. And I suspect many of you might not have actually seen it. So I'm going to go ahead and kill the port that may have been using that command, if it's live, and it looks like it. Yes, it is. So you can ignore this part if you are seeing this demo later on. And what you can do is 56078. Okay, great. So it looks like that our server is not running. But what I can do next is I can run step eight, wait, no, it's not step eight, step nine. And what step nine does is that it runs the server locally, and it does that by calling you the corn, which is an as the server, it's saying, go look into the main file and start the app, an app called app, and then have an all refresh. And so if I go to localhost 8000, like I'm just going to copy paste this link, and go back to my computer, zoom it in, and then go to this website, you can see that it is actually serving the data that I wanted. So that's really wonderful. And if I actually go to the other endpoints, like say most common, which was which I mentioned, you can see that it looks like Pfizer is a very common one in Europe, with the with the exception of Lichtenstein. And then we also have the least common one. And it seems like also, yeah, it looks like there's a pattern, where perhaps J&J is the least common or unknown types are the least and Lichtenstein is that Pfizer BioNTech. So really interesting data there, I wonder why all of this is happening. But at least now we can we can take a pause and let's just review what we've learned so far, before we go on to the next step. So let's review really quickly. Remember that we talked about getting the vaccine percentages by executing a command like this. We also created APIs to show the data. So there was the data, then there was most popular, I renamed this to most common. And then we serve them locally using UV corn. And then finally, navigated to the local host to see the results. So you can see this is me, I'm connecting to my own local computer, which is connecting to a database. And my own local computer is running a fast API. So again, that's all good. But let's take it to the next level and launch this website actually on the web. And I'm going to do that using Azure App Service. And App Service is just one of my favorite things. So it's Postgres flexible server. And I'll tell you a little bit on why. Now, if I want to package all of this information and put it on the web, all I have to do is run this command. That's it. Easy web app up. And because I want to be specific about the name of my site, calling Europe COVID vaccines. But if you remove this flag entirely, it would generate a random one for you and you wouldn't even have to do that. So easy web app up what it's doing right now is that it's taking all the code in my repository. It's and is going to zip deploy it to Azure. And Azure is really going to make sure that everything is up and running for you. So you don't have to provision containers or anything like that. It's really going to install your requirements for you. All of that cool stuff is going to be done automatically, which is really really cool. But while this is happening, remember that we're still going to need to connect this website to to to that Postgres database. And remember, I once asked you to save this connection string. And I'm going to tell you where this is going to come in useful. I'm just going to go ahead and copy paste it. And I'm going to go now to the Azure portal. So that's portal dot azure dot com. And I am going to zoom in very quickly. And I'm going to search for something called app service app service and Azure web apps. It's really the same thing. So I'm going to go ahead and click it. And I'm going to search for the app that I just built, which is called Europe COVID vaccines. Just because I have a lot of apps and you can see that it says that it's running below it is still deploying. And what I need to do now is I need to configure the parameters. And to do that, go into the settings and click configuration. And then within the configuration, you're going to see this sign on new application settings. So let's click that and call it connection string, and then put the entire string that we had over there. Sorry, hang on, I'm going to hide this and then click OK. And then click save. It's very important that you click save once you're once you've done updating the setting. And then the next thing that you have to do is you actually have to set up the server that is going to actually run your app. And app service uses Gunicorn, but you can specify the class and specify it to be UV corn. And so what I've done is that I'm saying that when this app wakes up, use Gunicorn with one worker, that's this flag, and use the class of UV corn workers and open up the main file and then execute the app. And once that's done, it should be able to run beautifully. I'm going to click save again. Very important to do that. And then click continue. And it's going to tell me that it's updating the app settings. Now let's go back and see if the code deployment was complete. And it looks like it was over here. It says that the website is up and running. And so let me see if we can browse it. And I'm going to go to step 11. And all that happens here is that I go to AZ web app browse, and it opens up things for me. And if I go hang on a second. Yep, if I go here, you can see that I have the website up and running Europe COVID vaccine dot Azure website dot net you can access it right now learn a little bit about the percentage shares in your own countries, or any country of your interest. And then you can even go to the most common and least common endpoints that I showed you. So with that, your back end is now live, which is really interesting. And now the next thing I'm going to do is do a quick recap. Remember, at this stage, the things were pretty simple. We ended up executing AZ web app up with a name. And then I set up a startup command in the Azure portal, as well as added a connection string. And then I went to view the website. And actually the one that I have right now is more pretty printed. But I use the non pretty printed versions like a squeeze all this information in in into the deck. So with that done, I am now talking to Azure app service, which is now talking to Azure database for Postgres flexible server. So at this point, I'm fully in the cloud, which is great. And now my app can see just about can be seen by just about anyone in the world. And so, you know, that's great. But now what I want to do is actually create a visualization so that the numbers make more sense to me.  And that is going to be in step 12. And all step 12 does is that it opens the visualization notebook that I have on my GitHub. And here it is, you can see that I'm making a request over here. I'm making it to a backup website that I had for this talk. But if you wrote your COVID vaccines, it's the exact same thing. You can see that the data is exactly the same from that endpoint. And what happens is that I picked a country that I'm very interested in visiting. That's Norway. I love the fields there. And so I wanted to know, well, I wonder what Norway is doing and what the distribution of the manufacturers looks like. So I went and I created the data and I got the country information for for Norway. And that was really cool. But I really wanted to visualize it because it just I like to see things in a pie chart format. And so I ended up making a pie chart out of this. And what I have here is the distribution for the vaccines for Norway, we can see that more than half the population seems to have that that's vaccinated seems to have Pfizer. We have a sizable amount of unknown types, which is 25% about a quarter of the population, not sure what happened there. And then we have a lots of Moderna AstraZeneca and GNG as well at five at seven seven and 5% approximately. So with that done, we are good in terms of the technical aspects of this demo. And I'm just going to review the last piece really quickly. Remember, I created a visualization by calling a backend API. And really what I had in the end was me talking to Jupiter, which was talking to Azure Web Apps, which was talking to Postgres, and really getting a nice pretty graphic at the end of it. And so so that's what I have here. And we built it using all these different technologies, which is really exciting. And I've put in all the resources, including how to deploy and web app and database together from the portal. I want to give a specific shout out to this because I know a lot of people don't like doing things in the terminal. So if you're a more visual learner or just a programmer, feel free to follow this link and try and doing this. I also have the code here, the readme instructions, as well as the data. And finally, I want to thank you all for being here. The one thing that I really love if you could do is please follow me on Twitter. I'm trying to build my Twitter app and also try and network with me on LinkedIn, comb the repo, play with the data, tell me what you've learned, tell me what your insights are. The code is available for you on GitHub. And most importantly, have fun. And it was so nice meeting you all today virtually. So that is it for me. I'm going to go back and talk to you now. Thank you very much for your talk. It was very informative. And all of us, we've definitely learned a lot. Thank you again.


What's up everyone? It's your girl Kinsey here from Layton Sports Cards and I am ripping 2022 Bowman Baseball Sapphire edition 10 box break number one random teams. Here's a look at the random result. You can always find these at LaytonSportsCards.com underneath the results tab and the link is also in chat for you. Alright best of luck everyone. Here we go. Again only spots left. Only breaks with spots left is going to be some prism football half case. That's the second one going and then full cases. Pick your team of flawless football. Let's get those filled up. They will be breaking after baseball. Alright good luck everyone. Good luck. Beautiful looking cars. Nandes, Martinez, Ed Howard and Pedro Pineda. Got Artega first. Giants. And Luke Waddell. First auto Braves. Braves taking home the first auto of Sapphire here. Braves on the combo spot. Brian D. Got Benny Montgomery and Avera first. White socks. Avera, Jackson, Charrillo first. Verrero, Aguero, Campos first. Tigers. Ellie De La Cruz. Nice one there for the Reds. Ramos, Montos and Sugeski first. Giants. Brambello first. Boston. Vina, Fran Aldui first. And Daniel Lopez first. Brady Allen first. Marlins. We've got Aqua 255. Jordan Walker. Cardinals. Adley and Darren Baker first. Washington. Brennan Davis. John Kenzie Noel. Nice one there. Cleveland on the first. Luke Waddell. Braves and Eduardo Lopez. Red Sox. Pack of box one. Braylon Haraba first. Milwaukee, Marco Luciano. Jordan Walker and Edgar Carro. Clara. Angels. Got Jeremy Vargas first. Curtis Mead first. Reyes. Nice one there. A little bit of a color match. Soderstrom and Ian Lewis first. Miami. Preciado. We've got Green Auto. Allen Serda first to 50 for the Reds. Nice green there. 36 of 50. You know, Reds. Justin H with the Reds. Got a Valera first. Cleveland. And Oswaldo Cabrera. Yankees. That was a very nice pack. Rivera. A Venezuela first. Padres. Vivas. And Valerio. Perez. Luis Gonzalez first. Orioles. Francisco Alvarez. And Jack Swinsky first. Pirates. Rosmar Quintana. Nationals first. Pedro León. Diego Valesquez. Giants and Genoves for the Giants. Nice. Joshua Baez. Cardinals. Jordan Lawler. Cooper Kinney. And Henry Davis. Nice prospect pack there. Henry Mendez. We've got Samad Taylor. Orange up next first. And Genoves for the Giants. Beautiful color match on the Orange. Giants is Chris E. Very nice. 8 out of 50. Heck yeah. We will mag that one for you. Very nice. Wilma Diaz. Renkona's first. Giants. Harnayas for the Orioles first. Martin Jimenez. Blue Jays. And Andrew Lara. Nationals. That was a nice box. Green Auto. Genoves first. Orange. Marcelo Meyer. We've got Kalil Watson first. There you go Miami. Nice hit Marlins. Venango and Harry Ford. Austin Hendrick. We've got Josuar Garcia. Phillies. CJ Abrams and Lizaraga first. Padres. Any singles for what? Sorry I just saw your comment. Pastrano first. Cleveland's. Nick Gonzalez. A Classe first. Mariners. There you go. And Carlos Aguilar twins. Another great pack there. Ramirez. A San Quentin. Cleveland. Nualbi Marte. And Rafella. Boston. Anthony Rodriguez. Giants first. Cavadas first. Red Sox. Pete Crow Armstrong. And a Luca Tresch for the Royals. Nice blue color match on the Royals there. Jeter Downes. Angelis first. Padres. Baxterum first. And Jay Allen. Hernandez Alvin Guzman first. We've got another orange first. Phillies. A Logan Cerny to 50. 49 to 50 on that one. Phillies. Another combo spot. Brian D. Doing well on the Braves Phillies combo there. Looks like an orange and Justice Thomas first red. Gotta we've got a green. Ellie Dela Cruz. Boom! Ed's Justin A. Sick. We'll mag that one for you too. My goodness. And Jose Ramos first Dodgers.  Man this bowman sapphire. Raze. Again everyone there's more Braves of this tonight with Forrest. I think he has two of these as well. Not sure if there's spots left or not. Do a refresh here. Check on some Prism. See where we are. Josua Garcia first. Phillies. Marcella Meyer. We've got a Ben Yemen Bailey first auto. White Sox. White Sox taking home this auto. Chicago and Harry Ford. Prism. Prism. Prism. Still six to go everybody. Still six left. Here is the link. Let's get those filled up. And most of those teams if not all of them are on sale. Half case break. Austin Hendrick. We've got first off what a 55. Braylin Haraba. Brewer's. Abrams and Lizoraga first Padres. Pastrano first. Salas. Jonathan Classe first. Mariners and Aguilar twins. Got Sam Quinnen. Cleveland. Noelle B. Marte. And Ruffella. Boston. Anthony Rodriguez first. Giants. Cavadas first. Red Sox. Peacrow Armstrong and a Luca Treche first. Royals. Jeter Downes. Unjealous first. Padres. Backstrom first. Graves. Carriotta Hernandez. Alvin Guzman first. D-Backs and Justice Thompson first for the Reds. Last pack box four. Nunez Alejandro Pia first. Reyes. Cespedes and Jose Ramos first. Dodgers. Alright box number five. And everyone baseball is all filled up for the day. We're looking to fill football. All football is pick your team. First half case break of Prism is sold out. Six to go on the second. And then eight, nine and ten left on flawless full case pick your team. So let's get these filled up. So we're ready to go when it's time to rip. Cespedes, Carriotta, Hernandez and Triana first Reds. Nunez, Pia first. Reyes. Carriotta first Reds and Dominguez. Armador, we've got Orange, Jackson or Jack Swinsky. Pirates to 50. Eight out of 50, Jack Swinsky. Pirates and Matt Frasier first auto pirates. Pirates with back to back hits there. There you go, Pittsburgh. Pull on my list again. Who had Pitt? Pirates. Rick E. There you go, Rick. Back to back hits there for you. Bobby Whit Jr. Garrett Mitchell. Robert Dominguez, Metz first. Couser and Ian Lewis first. Marlins. Sanchez first. Marlins. Vargas. Benjamin Bailey, White Sox first. And Will Wagner for the Stroves on the first. Rodriguez, Acosta, Vera first. White Sox. Panada, Elijah Tatees. You go White Sox, nice one there. Muziotti, Phillies first. And Benny. Dustin Harris, Rangers. Robbie Martin Jr., Rockies first. Salas and Robert Campos, Tigers. All right, halfway through and some very nice cards coming out so far. Hernandez, Artega first. Giants, Campos first, Tigers. Alcantara, we've got Chirillo first. Brewers, Sugasti first. Giants and Lopez first parts. What first out of that pack? Ramos Matos, we've got Ricardo Genoves. First auto giants. Man, giants crushing it. We had him again. Giants was... Chris E., very nice Genoves. Like that. And Darren Baker first, Washington. Oddly first. Ooh, we've got Purple. Ellie De La Cruz, we'll pull from the back. Adley, let's see. Purple, show me a first. First for Pirates. Daryl Lopez, 14 of 25. First Purple of the day, going to the Pirates. Rick E., jeez. Pirates eating up there. Thank you. Brady Allen first, Marlins. Brian Bellow, Red Sox. Irbina and Eduardo Lopez, Red Sox. John Kenzie Noelle, you know, Cleveland. Luke Waddell for the brave, seen a bit of him today. Trey Sweeney, there you go. Yankees first one of him, I believe we've seen. And Carol, Angels. Luciano, Walker, Brennan Davis and Shalyn Polanco. Nice crossback pack there. Last pack box, six. Luis Verdugo, Cubs. Austin Martin, a Braylin, Haraba first. Brewers and Jose Rodriguez, Rangers. Where we are on Prism. Refresh, lots of spots coming in.   Donras. Still six left, everyone. Again, there's some great teams left. Most of them are in sale. We've got Panthers on sale for 79 bucks. Lions for 99. Vikings, 149. Patriots on sale for 1500. Jets on sale for 700. And Steelers for 249. Again, that break is coming up in just a bit. Still six teams to go. All of them are on sale. There's a link if you would like to check it out. We've got Artega first, Giants. Fernandez and Campos first, Tigers. Tara, we've got Jackson, Chirillo first, Brewers. Sugei, Pirates and Darren Locaz first, Pirates. Ramos, Matos. We've got Christian Gonzalez first auto for the Astros. Astros taking home an auto. And Darren Baker first, Washington. Irbina, we've got Fran Aldui. Cleveland. A. L.E. De La Cruz for the Reds. And Ali Retschman. Luke Waddala, we've seen a lot of him today. A lot of this. Braves. Brady Allen. Marlins. Brian Bellow. Red Sox. And Eduardo Lopez for the Red Sox. John Kenzie Noelle, Cleveland. We've got Aqua first. Payton Wilson to 55 for the Royals. Nice hit there. Royals. Casey. Clinton I with some color. Six to 55. Trey Sweeney. Yankees and Edgar Carroll for the Angels. Luciano, Jordan Walker, Brennan Davis and Shailen Polanco. Crossback packet. Last pack. Luis Verdugo, Cubs. Austin Martin. A Braylon Haraba. Brewers. And Jose Rodriguez first. Rangers. Here we go. Benjamin Bailey first. White Sox. Mitchell. A Robert Dominguez first. Metz and Escato. Pirates first. Junior Sanchez first. Marlins Vargas. A Max Muncie first one of him that we've seen today. There you go. A's and Matt Frazier first. Pirates. Let's see a red. Let's see a red. Emmanuel Rodriguez. Maximil Costa. I'm pulling it back here. Nolasco first. Pirates. We've got Aqua. First to 99. Otto of Braylon Haraba. Brewers. 76 out of 99. There you go. On the Aqua Otto. Brew crew there. David F. Nice. Muziotti first. Got Mauricio. And Gonzalez first. White Sox. We've got Orange. Rockies. Hey Benny Montgomery. 35 out of 50. There you go. Rockies. Colorado. Who had the Rockies? E&C. There's Benny there. Robbie Martin Jr. Rockies. Jose Solis. Elijah Tatis first. White Sox. Eddie's Leonard first. Dodgers. Montero first. Yankees. Dustin Harris. Rangers first. Veneta and Will Wagner. Stroze first. Casas. Lewin Sosa first. White Sox. La Brata first. Mariners and Vera first. Martinez and Howard. Peyton Wilson first for the Royals. And Benny Montgomery. Solid box. Box number nine here. Cresciato. A Vargas first. Brew crew. Curtis Mead for the Rays first. And Aguilar first twins. Venezuela. Padres first. Vivas first. Dodgers. Vivas. Lizaraga. Padres first. And CJ Abrams. Alvarez. Vera. We've got Green Auto. Felix Valerio to 50. Brewers. Nice big auto there. Out of 50. David F. With another big hit on the Green. Oh my god. And Harry Ford. And I got a box of this in yesterday. This is making me want to rip it. Perez. Luis Gonzalez first. Orioles. We've got Orange for the Rays. Elberto Hernandez. Nine out of 50. And Quintero first. Cubs. Quintana. Nationals first. Pedro León. And Diego Velesquez. Giants first. And Valera first. Cleveland. There you go. Guardians. Joshua Baez. Cardinals first. Jordan Loller. Jack Swinsky. Pirates. And Felix Valerio first. Brewers. Martin Jimenez. Blue Jays first. Henry Mendez. Brewers first. Samod Taylor. Blue Jays first. And Genoves first. Giants. Nice first pack there. It is Brennan. It really is. Cerny first. Phillies. Rincona is first. Giants. Renea is first. Orioles. And A. Cooperkin. All right. And Lafayette. All right. All right. All right. All right. All right. All right. All right. And last box, everybody. Such a beautiful product. Again, all baseball is filled up. Let's start getting football filled up, everyone. Again, we've got two half cases of prism football. Pick your team. The first half is filled. Looking to get that second one. Last time I looked, there were six teams left. And then quite a few teams left for flawless. Those are full case break. Pick your teams. Three of those to end the day. And Howard. Peyton Wilson. First Royals. Crosses for the Red Sox and Panada A's. Fernandez or all of us Martinez. Benny Montgomery. And Vera first. Wengsauce. Cheerio first. Brewers. Piguero. Artega first. Giants. And Capos. Tigers. Ramos. Montos. Alcantara. And Daniel Lopez first. Pirates. Urbina. Aldui. Guardians. Eli De La Cruz for the Reds. There you go. Sinsi. And Sugaiste. Giants. Brian Bellow first. We've got Aqua Luis Verdugo for the Cubs. There you go Cubs. Very nice Verdugo. The Cubs. Clinton. I. Adley and Darren Baker first. Washington. Select baseball. Yeah, they just had a photo. What was that yesterday? Or the day before? John Kenzie. Noel. Cleveland. I was out shopping and I missed it. I was probably going to snag a boxer too. Luke Waddell. Braves. Brady Allen first. Marlins. And Lopez first. Red Sox. All right. Last pack everybody. Walker. Brennan Davis. We've got Aqua. Let me pull from the back. Carol first. Angels. And last card of the break first for the Guardians to 99. George Valera. Boom. Guardian. There man. 53 of 99. Fantastic way to end this first break of Sapphire. Wow. We will mag that one of course. All right. Give me one second and we will do a recap here. All right. We had Harabata 99 Brewers. Christian Gonzalez. Astros. Matt Frazier. Pirates. Benjamin Bailey. White Sox. The Alan Serta on the green to 50 reds. Luke Waddell for the Braves. Then our mag cards. We had the green first of Felix Valerio to 50. The Ricardo Genoves first. Giants. Green first of 50. Ellie Dela Cruz. Orange first. Genoves to 50. And then the Valera. Aqua to 99. Congrats again everyone. Thanks so much. We've got another Sapphire coming right up in a few.


Hi, thank you for attending our 5.30 presentation today, realize it's late, but you know, just be patient and then you can go have the drinks outside for the happy hour. So today's presentation is about tuning OpenStack for availability and performance in large production environments. And my name is Gabriel, I work for Symantec and I'm part of the cloud platform engineering team and we're building a consolidated cloud platform that provides platform and infrastructure services for the next generation of Symantec products. And a little bit about myself, my background. I have more than 10 years of experience in large scale production environments and I've been doing OpenStack from probably 2011 for about three years and at Symantec we're doing that and I'm part of a great team and really excited. This is our agenda today, we don't have time to cover all the topics we wanted to but here are some of the topics that we're going to be covering. So we're going to start with talking about what large scale means for us and what high availability means for us. We're going to talk a little bit about infrastructure lifecycle and how we deploy and provision boxes and also about configuration management and some about orchestration. The next topic is going to be about how we've integrated Keystone and the art services with our enterprise directory which is LDAP. The next topic is going to be about Keystone and securing Keystone and about PKI tokens. My colleague Raj is going to come in later and he's going to talk about NOVA and some of the KVM tunings and also the database tunings and lastly the RabbitMQ cluster. So let's get started.  So what we consider to be a large scale. So first of all it would have to be spanned through multiple data centers. It will have thousands of hypervisors, tens of thousands of VMs and millions of requests per minute to your API endpoints. And obviously these numbers are, they will vary based on the sizes of your hypervisors or your VMs. In other words if your VMs are very large VMs you're going to have a lower number, same thing with the hypervisor. A hypervisor that is 12 cores and 64 gig is going to be a lot different than one that has 40 cores and 2 terabytes. So these numbers will vary. What about availability? That is high availability and we're talking about the control plane here. So we're aiming to have four nines of high availability for our control plane and we think we can achieve that number and the way we have done this is to first of all our control plane is virtualized and is distributed among failure zones. So we don't have anything running in a single failure domain. We use hardware load balancers that come in pairs between, in the front of all our services and the next rule is about the separation of your network. So I don't know how many people here have had issues with spanning tree because, okay, so I've had some issues as well and I remember those nights. So spanning tree is good but the problems that come with it are really ugly. So the rule is no spanning tree across availability zones and so each one of our availability zones is single L3 domain. The next thing is you should go with redundant power and redundant network connectivity and pretty much everything redundant as much as budget allows you to do so. And to give an example, I put here one of our hypervisors, how it's configured and as you see we have multiple redundant 10 gig nicks and so 10 gig nicks today should be pretty much on board on any infrastructure, on any enterprise class hardware server and the cost per ports has come down and it's probably not going to go up. So you should have redundancy where you can. So as you can see we're running in an active, active mode with LACP and we're sending multiple VLANs, we're running trunk ports and we're sending multiple VLANs to the hypervisors. Again these VLANs do not span between racks so they're completely isolated to the rack. And also very important have your management interface completely separated so you can always go and, if something goes wrong you can always go and manage your system there. So the next thing is infrastructure lifecycle and how do we provision boxes and how do we manage these systems. At this time we're using Foreman to provision our boxes and at the end of the provisioning we have, we're classifying the systems in different classes and then that classification ends up being used by puppets and for example if it's a compute node or if it's a storage node puppet applies different sets of modules and about puppets at this time we're using, we have switched to a master list puppet setup, we have used the puppet with regular servers in the past and although we have had some issues with scalability of it, that was not the reason we switched away. I mean there were many reasons but I think mostly because of it gives us more flexibility. Well our puppet code is in Git and we have a very easy way to pull these modules and apply them and the other thing is we can do so on a very large number of systems at the same time. Previously we were having issues running puppet job on let's say 2,000 boxes at the same time. That was a problem. Puppet servers were having a problem doing that. So without puppet servers that's not a problem anymore and we use, for orchestration we use salt and fab as well. A little bit about enterprise directories and OpenStack. So you build a cloud and you want your users to start using it. So what are these users? Users are if it's a large production environment your users are somewhere, they're either in some sort of active directory or they're in an LDAP directory. In our case we had both so now the question is which one you use. Active directory there's nothing wrong with it, it's just that in most of the enterprises or organizations it's usually handled by IT, it's owned by IT and you know our IT department is great and they're doing a really good job, it's just that you know their priorities are sometimes slightly different than ours, our job is to keep the site up and their job is to have our computers and our email and everything else working. So when you have a problem you might have to file a ticket and you don't want to do that.  LDAP is perfect because it has built in replication, it has built in synchronization of data, it's the perfect directory for read-only queries which is exactly what you want, that's exactly what Keystone does. So that's why we went with LDAP and the other thing is you can, because you control the infrastructure you can design it so it fits your needs as far as load and performance as well. So at this time we have used LDAP only for identity. We kept the assignment in SQL in MySQL like stuff like tenants and projects and domains are still in MySQL. So LDAP is a read-only directory so people sometimes make it read-write and they think they should go and create users from Keystone and don't do that. Leave that problem for something like identity management or pretty much everybody should have a system of getting their users onboarded and don't get into that. So use it as a read-only. The next thing is the security of disconnection between Keystone and LDAP. Make sure it's using LDAP S and I've seen sometimes people use just regular LDAP on Portrait and I'm thinking that this is going to be TLS but the truth is TLS sometimes can be, it cannot be enforced all the time so if you're just using LDAP S you're sure that your credentials are going to be send encrypted. Also work with your LDAP administrator, create a special user for Keystone to bind to LDAP and set the right permissions for this user to allow this user to access only the rights or use and that will give you a lot of leverage when you have problems with the LDAP directory and your administrator will be able to pinpoint these problems. This is a diagram that shows what I've said before, the end-to-end encryption of your credentials and my colleague yesterday had a presentation on Keystone and she brought some of these things up already but make sure that all the links in the path of the credentials are being addressed. For example, how do people talk to Keystone? So one way is through a GUI, through Horizon for example, another way is through Keystone Client or through CURL Direct API calls. So you want to put, first of all you want your Keystone endpoint to be secure so use HTTPS there and then Horizon as well, HTTPS on your Horizon interface and maybe like a redirect rule that will have people use it. And we have already talked about the connection between Keystone and LDAP that is also encrypted so by doing this you are sure that at any time the credentials are protected. This is another picture that shows more detailed diagram on how the security on Keystone has been implemented. So as you see we are using load balancers and we have split the LDAP, we have split the Keystone and the Keystone admin into two VIPs because we wanted to use, we wanted to take the ports out of the URL to make it easier for everybody. They just say HTTPS, Keystone and the rest of the domain and that is the endpoint. No ports. So there is a port translation happening on the load balancer and as you see we are using on our Keystone servers, we are using self sign certificates versus the trusted sign certificates on the user facing ones. And somebody asked yesterday why do we do this encryption here since the data is already encrypted. And the reason is sometimes the compliance, because of the compliance rules if the information is passing from a more secure zone to a more secure zone through different other zones the information needs to be encrypted. So it was not a big deal. So we used Apache with the Mod W, HGI and Mod SSL, self sign certificates then on the load balancer client SSL and server SSL profiles and the discussion of which certificates you should use signed or self signed. Remember it's always about user experience. You want your users to not feel any difference when they come and use your cloud then they go and use a different public cloud. Like I haven't seen anybody complaining that I've got a warning certificate mismatch when they go to use a public cloud. That's never going to happen. So why should it happen for yours? So always customer facing VIPs use trusted CAs. So we're going to talk about Keystone and PKI tokens next. So I was curious how many from here have been using PKI tokens or are using PKI tokens? Okay. So not very many. So the question is why should I use PKI tokens? Why should I just not stick with UID tokens? Because you know they work, there's nothing wrong with them. The problem with UID tokens is that when somebody requests a token from Keystone they authenticate, they're getting this token. This token will have to be sent to a different component and it will have to be validated by that component. And then that component will have to contact Keystone, Keystone will have to validate the component and then it's going to have to reply back with OK or a list of more information. So every time that happens Keystone is being queried. When you have a really large environment, Keystone is going to suffer because of this. So then you switch to PKI. So PKI what's happening is Keystone sends you this token in an encoded format, it pretty much takes the JSON file that has this information and it encodes it and then it's signing this information. So the client pulls the certificate that was used to sign the token and it can validate the token without contacting Keystone again. And that's very important because it pretty much takes the score out of the, it never talks to Keystone again. There's some misconception there about, oh I switched to PKI tokens, my tokens are very secure and nobody can see the tokens. The tokens are not encrypted, the tokens are only signed and they're encoded. Anybody can take the token and decode it with base 64 and they can look at it. So remember that is tokens are not encrypted, they're signed. So there's a certificate expiration, there's a client that can validate the expiration on the signature and then make sure you, there were some problems before with their vocation list but they should be fixed by now. So it's not all good with PKI tokens because otherwise I guess more people would have used them. So there's some problems mostly related with, for us we have had some issues with the Keystone catalog size. So for one reason or the other our master catalog size had grew more than it should have. And sometimes when people request a scope token, this catalog has to be encoded into the token. And what's happening is we have seen that because of that this token gets passed out in the header of the HTTP request and a lot of components have issues with larger header size. So one of the options that you need to make sure it's being looked at was this Max Sederline option that I think it has been bumped from the default 8K to 16K in some components from what I remember but to be sure we have bumped it to 32K. And also, you know, newer mod WSIG has a specific option to deal with this. And also very important, talk to your users and have them use no catalog when they request a token if they don't need to have the catalog in the token. Sometimes they do not need it and they don't know about this option so that they can request it. And I think at this point I would ask to keep your questions for the end when we're going to have a Q&A. And at this time I'll have to let my colleague Raj to come and talk more about Nova and these other things. Thanks. Thanks, Gabe. Hi. My name is Raj. I work in Cloud Platform Engineering in Semantic. My background is on operations and infrastructure and I've been working with OpenStack from the past like two and a half, three years. So one of the important aspects before I start is like I want to ask people how many of the people here using Nova API behind Apache? Okay. Not bad. So one of the crucial important things that we saw related to performance especially is there's nothing wrong with Nova API using existing Python evenlets. But when you get lots of requests and especially Gabe mentioning, my colleague mentioning about the PK token, the header size increasing, you're definitely going to hit the performance and most of the things that we see is like, all right, this performance issue let me scale the worker. And you can't, you keep on scaling them but you're not going to get the performance better than the throughputs from the API services. So Apache has been proven through these web services well, better handled. And the other reason also we thought to use Apache because we want to make sure every layer of OpenStack is being encrypted and use SSL in the back end. So today we don't use that SSL for Nova but we have plans and if you see here we have commented that out but we're working on that. By using Nova API behind Apache, the differences that we saw being eight workers with Python evenlets and with three processes with Apache is going to serve it. So you see a significant difference. It is test config files that we use today. One is for Apache configuration to make the virtual host and to enable the ports. The other is for Apache, more WS, SGI, the script to invoke the Nova API service. Feel free to use this and then try in your labs and see the difference and you'll definitely see a significant difference. The other thing is I'm not going to touch more on the Nova scheduler because we use our own custom scheduling and most of the people I guess are also using their own custom scheduling for their own workloads. So I'm going to skip to Nova conductor. How many people doesn't use Nova conductor? And I think everyone uses. It works for you guys. One of the important aspects about Nova conductor is it's been recently added in Grizzly. It's a great idea to bring security. It's been added because to bring security in place. Not Nova compute talking to database directly. That's one of the major aspects that it was being introduced. Saying that but still Nova compute has been compromised with MySQL access because if you're using Nova API metadata, you still need MySQL database access or if you're using any volumes like Cinder, it still needs MySQL database. So apparently it's not solving everyone's problem but it's a good idea to start using that. But we see some performance issues by bringing Nova conductor into the picture. People say that yes, you can mitigate this problem by scaling it out but the same time I'm scaling out my database and I have to scale this out for no reason because I'm not getting an advantage today. Maybe I get advantage tomorrow. Maybe I should think about it. So to mitigate that and to get better performance, we actually disable Nova conductor. That's how it looks to flow. It's not the default behavior anymore. You have to put a configuration option in Nova.comput to disable that. And I'm pretty sure that you're going to see significant difference in your performance because you're not going to have one more layer. Because if you see Nova conductor talks to rabbit message and then talks to database and you will have massive issues of bottlenecks with this and also have one more step. So to do that in the config file, use an option called use local equal to true and then our compute. People who doesn't have MySQL configuration, make sure before you use this option, configure MySQL options like put your MySQL web using a password and database information there so that you're not going to get added out from the compute node. And this is one of the important things that we want to share. And also we're trying to actively work with the community to see what is the best we can do to scale out and maybe better idea for securing the database access. So the next important thing is also, we said it's performance tuning on OpenStack and it turns into KVM is kind of tightly being integrated with OpenStack nowadays because the only people by default when you say OpenStack, people also point to the KVM for virtualization. We did have done, I'm not going to talk about a lot of performance tunings in KVM but I'm not going to talk everything here. But we want to share a little bit what we have done, which are the major components that gives value, adds value to your existing cluster of workloads. One other thing is KSM, channel same page merging. This actually helps you to get, to share your identical memory pages among different processes or virtual guests in a virtual host. It's into a single virtual host. This is actually also so critical if people are doing their resourcing, like the provisioning resources is like overcome and made like having increased their VCPs and memories, especially memory, will help a lot. And also, this is not a tough thing but it's about the thing that we might be forgetting to install a package and turn the service on. There are services that take care of this. So in Rull based versions, it's KSM Tundi is a process, make sure it's turned on and it's an Ubuntu KSM D. Make sure it's turned on and it'll follow. The process will take care of the rest of the following steps that it's going to take here. I'm not going to go much details there because it's a straightforward thing. The next important thing is transparent huge pages. I think everyone talks about it and it's so important because it also gives benefit not only host but also guest. By saying that, you have to also keep in mind that the value of the host should be always, the value of the guest should be always less than the host because you're going to not gain much performance if the value is much increased. Also, to make it enable, it is a command to make it enable but if it's always disabled. If you cat on that file, you see it's never, so make it always to make it enable. The other thing is also add this configuration parameter in your Libboard XML file for a guest operating system to take advantage of it and there are other parameters that you can tune in. We're not really sharing the number here what we're using because it's kind of a test and tune. It's based on your workloads, so you can use the default. Maybe some people use the default value which was great for them, but some people may not. So the next thing is blockhive, the IOS scheduler especially. One of the major important things from the default behavior, CFS, we moved to deadline. It gave us significant performance increase when we use QCock images especially. It doesn't match with the raw but still it's a little bit performance increase by doing that. Also keep in mind and also if the same team dealing with the guest and the host, it's okay if there are two different teams that are dealing and then first exercise, make sure that coordinate because the value is being already configured at the host level and if you're trying to do on the guest, you're going to have a worse performance rather than having a best performance. So the other aspect also we have been seen and also we have been beat by it is because enabling cache equal to none, make sure because it's not default in the OpenStack, it's not default in the config XML file and I will equal to native adds more value. The reason because you already have a cache in the back end of the hardware which takes care of the things and you don't want to do again which gives you worse performance. That's more about the major three things that we actually looked at and we want to share that in the configs and the comments about KVM. The next one is database. It is kind of tricky because database is always a pain point in every aspect. The thing is people talk about like how do I replicate, do I have to do multi-master, master slave, how do I do reads, how do I do writes. So one of the approach we take is we use a standard model of Galera application. We have a three node cluster and it's not always ideal because based on your size of the and your workloads you can always increase and make sure it's odd number always for the speed brain mechanism. So we use WSwrap which is a write set replication. We enable the data replication behind the load balancer which is taken care of and we have a web that all clients connect to the web. These three nodes are all active and we try to make it to write to all nodes and we had problem and then what we start is like make one node write, make sure that it's committed to both nodes and then read from all the nodes and so we're differentiating the workloads between write and reads to get better performance. Talking a little bit of Galera Galera is a data center approach that means every node has its own unique ID and the rule of thumb here is the node belongs to data, data doesn't belong to node, that means data is not part of node, it exists everywhere. So no head cache or increment as legacy MySQL replication does here doesn't do that anymore. So we want to share our configuration here too because it might add value for everyone. What are the values that we do for our cluster with three node? How does it increase? So we have been like testing with the massive workloads and increasing each individual performance for each individual configuration option and then tune them to our workloads but it also depends of your workloads but this works perfect for us. Feel free to use it and test and see. The other thing is there is always software, we always have limitations, everything is not perfect. So the most important thing is it supports only InotDB and make sure that your database has been using InotDB and primary key is masked. With our primary key please do not even try using Galera cluster and that can be fixed, the schema can be fixed if you can spend some time on your schema which we have done a few times. And also commit latency which is another one that we said like when we make it all active that's what we actually ran into problem with commit latency because of different workloads and reading and all these things. So that's the reason we made one master and then the rest like it's a pristine connection behind the wave. One master gets all the connections and then when it fails down then only it goes to the second or third masters and we do have a custom scripts behind the load balance that checks whether the node is in the cluster or it's not in the cluster. We have a separate user that has been created to MySQL that we use that to check that nodes. The other thing is it doesn't like huge transitions, it doesn't like to put really huge transitions and deadlocks and commit is always a problem with all the databases not only MySQL but it's one of the limitations. That said this is what or few limitations there are few more but these are the major limitations that we see from our side. That said now we're kind of switching to Rabbit MQ, this is one of the coolest things so I just wrote it down. Rabbit used to be using for cartons and now it started using for data queues. So Rabbit MQ, what can Rabbit MQ do for us, for you, for everyone in the open stack cluster? The one of the important aspects people always forget is there are a lot of differences between Rabbit 2.x and 3.x. We use 3.x. It's been improved a lot especially it has an A2 clustering mechanism only to use a pacemaker and chorus sync which I saw a lot of operations guide this but you know to do that we try to contribute that to have this documented there too and it has a high level queues that means by default the queues are not part of the cluster. We have to enable the HA policy on the queues. Don't forget that but that's that's one of the most things. People create the clusters but forget to run that command and also it has the latest implementations for the MQ protocol 0.9 and 1.0 came recently which has a lot of cool features especially connection pooling for connection channels and federation is another important cool thing that people can use it if you have an isolated networks that you want but you want to send the queue in different places you can use federation you can set a policy that xq go to winord and then it goes and Rabbit take care of a good job there and flexible routing for the queues so when a producer puts a message to the exchange and then you have a flexible routing that you can enable what kind of routing mechanism you want to use between exchange and queue I'm not going to go much details with Rabbit subject here it's a it's a big one so this is how our cluster looks like so we have a cluster with three nodes here so the cluster automatically replicates all the exchange and it's on data and maintains a lot of database and we have a queue that is that that got created in node one and got replicated here to node two and node three and this is the same thing when a kind of connection came here and if you create a queue in node two it's going to get replicated node one and node three so this is how our configuration file looks like and we would like to share that too because most of the implementations in my experience that I saw people run commands using their configuration management but Rabbit pretty much you can put things in the configuration file stay statically so that you know to run or forget and go with your craziness of configuration management so you can set once by every restart it's going to read this configuration file and then go with it the other important then I said like you have to enable the policy to replicate the queues I mean to have a mirror the queues so here's the command this is what we do so usually what we do is we usually add this command to the startup script itself because it's not going to harm anything if there is a policy already set it's not going to do anything if it's not it's going to set the policy we're making sure that it exists and usually when you plan for cluster when you create a first node you can run this command on the first node so that one that node two one node three joins automatically the queues will be mirror so so that's that's that's where it is so one of the other important thing that I want to share why why we are using a VIP before the rabbit cluster people might be asking OpenStack has an implementation having a HEA queues that you can put host a list of hosts it implements this with a for loop going node one or two node three it always goes in a loop node one node two node three and we don't want to use that reason because it's a client implementation it's it works great but at the same time I'm not going to get get it load balanced I'm getting HEA but I'm not getting load balanced so I want to use load balancing functionality the rabbit so that I get better performance by doing that I'm not choking at one single node because what happens when you have thousands of hypervoiders assume node one goes down all the nodes are going to be node two it goes down all the connections are going to be node three and you have a choke point there some people have seen the confusion management puts a logic odd number hosts go to odd number node even number goes to still it's it's how I wonder but we use load balances because we do have a little bit of load balances and we do and we also use connection caching on the load balances for connection pooling to give even better better performance and also make sure that you have keep alive enabled on your load balance it's a little bit the degree of performance but it's not that bad that's all about about robin mq and thanks for listening to us and any questions so you mean which manufacturer yeah so the question is what load balance are we using so at this time we're using the 810 but any load balancer will work just fine can use f5 or citrix it's not specific to a vendor I mean to add to that we have not tried hf proxy we're planning to try that too but we have an enterprise usually you have hardware load balances so we're taking advantage of it we are in havana and we are trying to upgrade to icehouse we have not seen anything like that from day one when we have set up this cluster we are using behind the load balancers and make sure to keep alive is there that's the reason it's there but we have not seen any that kind of things we see some of the timeouts losing connections from the consumers and it's a it's all about the one of the thing that open stack does with the loop is keep on check but it's all about the client implementation right how do you keep on check and connect back that's the reason we have a keep alive enabled actually to get indicated that but one of the thing that once we notice that we actually did little hack I would like to say because we use that configuration option on the config file of every component we use the same whip same whip repeatedly so this connection last first one still it going to try and try so we have not seen any specific RPC timeouts and this is the single cluster that talks everything even our SDN and everything so they have not seen we have we for that we actually use to add to that we use stack stash which is a rack space project for listening to the notifications we do track of that we have not seen anything like that because we keep on track every every minute what's going on on our cluster notifications and we track the flow of it's going and that is the problem but we have not seen anything we see okay with my sequel are using load balancer introspection to actually split your rights from your reads how are you directing your rights specifically to one node so we have tried the we have used that in the past with f5 you can write an iRule that will inspect the sequel query and then it can direct the traffic to specific nodes and you know where it's kind of makes load balancers a client server kind of thing so that's the reason it needs an sequel user to make sure that that what kind of commit it's coming up it's an update or to see what kind of sequel query that it reads and then based on the it takes so you are still using that on the a10 to say these are rights and we don't we don't use that at this time okay not at this time that's the reason we have made enable the persistent connection to one node right now and leaving that but we're focusing to go in the direction okay cool so you guys cut out the nova conductor yeah the community is really against the computes having db access yes have you looked into increasing the efficiency in conductor versus cutting it out good point actually even as we are security company we are concerned about that but conduct is not giving us value uh in terms of even uh even the community is saying that it should not have access to my sequel but still i need my nova api metadata needs my sequel access i'm not it's not solving the problem so rather than spending time in uh throwing there because it's not ready yet we just cut it off and we're probably gonna go back and because we need that security to be there so you guys are using nova networks still no i think that's the end but it's also not only about that but also if you use cinder or anything it requires my sequel connection from the company so still it's not solving the problem all right thanks uh so the question is how do we split the so as i said our control plane is all virtualized and we have a separate domain failure domains for the control plane and so they're distributed among different racks and so they run on different hypervisors so we're making sure that they don't run in the same rack and they don't run on the same well they cannot run on the same hypervisor but if there's many services of the same kind in the same rack they won't be on the same hypervisors so uh one important thing is uh our workloads of control plane is all virtualized we don't we don't run on bare metals yeah so it's because the scale out it's cloud i wonder like you took on dogfoot one more question about rapid mq yeah i didn't follow did you say that you are using rapid mq hosts parameter yes to tweak okay yes because of the timeouts because uh that we have not fixed that still it goes to the whip but it's going to do the same whip and the load balance is going to take care of how the connection goes and one thing guy didn't mention before and they should have was that we have uh checks for the service for all these services on the load balancer so all these services are being marked down if the service is not performing so it's not only that the host is being able and the tcp port accepts the connection you also want to make sure that the service performs and then the law if it doesn't the load balancer mark marks it down so the client will never see the make sure the service is enabled and it's accessible by the user that's that's how we mark it up in the load balancer okay thanks the question is do you monitor the q and the node status we do monitor the node status we don't monitor the q but we monitor as i said by using stacky the flow of the q for each and every request that is going on the request ID has everything so i will definitely as you said if i see something is missing there i definitely see that the notification has not been there so that we see that but we don't see and we don't monitor that we don't go into the looking at the q but we look the notification and get the q and we save that into database for looking the history how many cues we i mean how many requests we served and how did it go what's the time that it took i think i actually want no more questions or you guys we can take one more question if you guys have anything okay or we can talk about the beach yeah we're gonna be around so if you guys have any more questions find us thanks for staying late and staying


He's about to get splattered ready three two one.    I couldn't see it till he lifted his arm. Oh my god. He got shanked Yeah, literally He is Oh, it's an S rank trust me, bro So just listen to the commandments So that's the god I think we're looking at a terrifying monster bringing this bad of a despair wait, we're going to school now Bro, this is like code geos We're like we're in the middle of war and then it just transitioned to a classroom Like man, I think there's more important things going on right now. Why are you suddenly talking about that?  He's mid. No, he's below mid, bro. He's terrible He's terrible, man. He's the one player in your team that just holds you back Like you'll play halo, you know the win the game. It's like to get to 25 kills Your partner will have 24 kills and you'll still lose because you have a genwoo on your team He sucks, but guess what? He's doing this for his mother the medical bills. He's doing it for her sister He's doing this for love and he's better turn into a K drama protagonist any second now. I'm sure it's going to be this episode What are you doing these people don't learn He's better get splattered ready three two one We watching that bro. The episode came out five minutes ago.  He got He got a race.  Oh You're not doing anything.  Oh god now.  They're gonna attack now. What is he doing?   This is cool. This is cool. Just follow the rules to the game That's it. There's no aim bots.  What do we do next song?  What's he doing here?  Bro that give me on the rumbling playbacks.  Y'all heard that shit? That was brutal. I want to die She's already moved on bro. I'm the other guy Why did park park? Why did park decide to just park right there? He got sliced in half bro.  What's her name? Look, he's taking charge right now.  They're not holding back. I'm loving it.  I'm not attached to this character. Okay. I'm gonna be honest with y'all That's canoe Oh, you gotta go you gotta each take a statue. Okay. Okay What is that the marshmallow guy Just wanted to get a pig. It's just gonna be a beautiful moment right now, isn't it? All something's better. We'll wake him inside of this is gonna be like an errant moment where he's gonna come back longer, isn't he?  This is orchestra Rada.  Did he lose the leg? Did he get shanked too?  Oh my god It's all right. It's gonna grow back grow back, bro Let's let's be completely honest. All right. I bet I I bet Every one of your lives that his leg is fine. All right.  You're gonna have a reaction.  I know what If you ask me, it's all at our fault.   He's not gonna let it be. He's not gonna let it do that Imagine this school to be in a footprint That wouldn't be that bad Time to attach us to the character That's the energy source, so we're skipping her alive Oh my gosh, he looks dead already No, bro, is she?   He's the smartest one here.    This is too good.   All right, so they're alive.   I would have bought more insurance. Oh god. Oh, this is the moment, man No, man.  They're gonna have to fuse.  I didn't even know that I thought it was like a flashback of the other guy I'm like, he's going through right now. He's feeling the pain of being ripped apart Oh, yeah, I'm done reading episode titles after this one I never noticed that that he was sitting he was laying on there.  This is his next chance, isn't it the blue fire. Oh my god That was the next episode. No, no, no, no, no, no. It's like the game the secret quest courage of the week Courage of the week. He's the week the boss is checked Okay, we've acquired the qualification to be a player.  Oh my god.  I'll do it right now Your heart will stop in 0.02 seconds.  You see what I mean? It should have been episode one and two in one That should have been one, bro That would have been perfect that sums up the episode like that that story that little that's like the beginning the beginning is over Now we're getting to the story.  Will you accept?  See I believe that I believe that Who is tk?  I feel like I didn't miss anything you miss skipping this show You missed opening with visuals. I already reacted to it before the stream I mean when the stream started I reacted to the opening. I'm uploading it to my channel. You just missed some dialogues well, was it like Insane dialogue that I missed because I feel like I got the premise I got like 98% of it I'm late, but have the dialogues changed to japanese names.  No, I don't miss that I've heard adult cover of ravel unravel. It was really good.  I really hope so episode one Was like introducing you to the world and I feel like episode two should have been attached to it But episode two is really when people will decide if they're watching or not And I feel like this is gonna make a lot of people say yes I'm going forward, you know if they didn't do what they did in this episode People would have dropped it because we have this tiktok brain now where people will give a show like one episode and then they will Leave I say give a show like three episodes and then you'll figure out if you want to watch it or not And this episode one was cool to me. It was cool. I forgot what I rated it But I said I wish it was more to it. This was more to it This gave more to it and this makes me want to watch more of it


My name is Kamalakshi and I'm studying telecommunications engineering and networking and it's my second year at the university. In the faculty, especially the faculty of engineering, it was a male-dominated field and girls were even scared to step into it.  But I think it has changed. When I joined this field of study, it was completely different. It's a very supportive and enriching learning environment that we've created where boys empower girls, girls empower boys and vice versa. It's just amazing to see that this has evolved and it keeps evolving. It is good to have a good gender balance because this is a field which is not restricted to one gender. The nature of telecommunications and networking has become more and more software defined. Most of the tasks can be performed directly by just programming applications. So it's good to encourage both male and female to be involved in this field because it concerns everyone. But we could do more to increase the intake of girls in our courses. And I think at that level probably the faculty and the department can do some outreach programmes at school levels to encourage students to join our programme. The job prospects in the field of telecommunications are immense. At the same time it's a very fascinating field where you can explore several technologies which are helping people in all spheres of life. And this is expected to have an even greater impact with the advent of 5G mobile technologies. Mauritius is in the eyes of others, a tiny dot on the world map that's barely visible. But with the major breakthroughs that we're having in the coming months we're planning a satellite launch and nano-satellites. We are moving forward. More girls are taught how to code in school from a very early age. And I think now is the time to be bold, to be brave, to break all the barriers and just go for it. If men can do it then I think women can do it as well.