Yeah, hello, hello, and welcome to another video. Video of me plus you is? Us, my name is Kwame. I'm Elaine. And we're here with another, yeah, we just said that already. Episodes. We, maybe for people who don't know us, we talk about things in our life as an interracial married couple in Ghana, Accra. And we just got a baby. Yeah. So we're a Dutch Ghanaian couple who just had a baby and we just share our experiences. Oh, yeah. And just a little bit context. We got the baby, or we delivered the baby in Netherlands because I wanted to be close to family and friends in the Netherlands. And we just got back to Ghana into our own house, which you see in the background, boop, boop. Yeah, so we know this is our house. I don't think they thought we're recording at the studio. We wish. You want to get a studio? But yeah, so in this video, we're going to share how it was like for us to fly with our baby to Ghana. And he was six weeks at a time. Six weeks. And we have to fly with him to Ghana. So we're walking you through the process. If you're watching this, you just had a baby. You're about to have a baby. Maybe you're also in our position. You have to fly to another country. Maybe you want to go on holiday with your baby. Good idea. Yeah. I salute you. Yeah, big time. OK, the first thing is prep all the paperwork. Yeah, the paperwork is very, very, very, very, very important that you have it all beforehand. And depending on the country, you're in a lot. Or the more the better. Yeah, so for us, it meant because we knew we were traveling quite soon after giving birth. So six weeks after the baby needs a passport. Otherwise, you cannot take the baby anywhere else. So before the baby was born, I had to plan an appointment at the municipality in the Netherlands to apply for the passport, which felt really weird to plan an appointment for a baby that's not even there yet. It feels almost like jinxing it. But I mean, just make sure you prepare ahead of time. And we also, in the end, applied. So we got the Dutch passport quite quickly, within three working days, which was great. But it was an experience on the cell because you have to make a passport picture with a baby. By the time the baby was two weeks old. Yeah. And we had to take the baby to the photo store. And I mean, it's a requirement that the baby's eyes must be open. They must not be crying. They must not be sleepy. And yeah. So you tried that with a two-week old. In the end, we managed. So they had like a pillow. So you had to put the baby on the pillow and then they would retouch the pillow. So it was on Kramis lab. And luckily in three times, three shots, the third shot was the right one. Yeah, it was OK. But it was really great that the photographer was very patient. Yeah, you could tell that he's been like, he'd been doing it for years because the level of patience he had was something we new parents were like, we wish we had that level of patience. Yeah, it was really like, OK. Very kind. Very calm as well. Then we were also calm because we were like, OK, he's not rushing us into anything. Yeah, then we got a shot. Yeah. So then we got the passport picture. And then if you apply for a passport in an ambulance, you need to bring the individual you're applying for. So we had to bring the baby. We had to go to another city. That was a thing, like to travel with a two-week old, pack him in the car, pack the food. So we lived about 45 minutes to an hour from. Where we had to apply. Where we had to apply. So we were in Haute and we had to go to The Hague. And yeah, we had to drive with a two-week old. Luckily for us, he travels very well. He likes that car motion thing. And we even got the opportunity to even spend a little time doing something like self-care for ourselves in the city after we applied. And that was pretty cool. But yeah, if you don't live close to where you have to apply for the passport, then you also have to now, for the first time, travel with your baby. Which means that everything takes twice as long. And your prep is also twice as detailed. Because if you miss one thing, like the pacifier, if you're going to be using that, how many diapers? Extra food. Extra clothes. For him, for you. So yeah, it's a lot. But it's good practice. Because in the end, I was more confident because we had stepped out a few times before we were going heading to the airport. So it also helps you to already get into that flow of, okay, we're stepping out. This is what we need.  Are they fussy? Do they sleep? So yeah, it helps us. Nothing is in vain. So it will also help you. And we even had to, so what we did in the end was also apply for the Ghanaian passport. That one was a bit tricky. Tricky because the process wasn't as smooth. Yes. And it would take longer. They weren't sure when it would be ready. So in the end, we applied for and the visa and the passport, which was a bit stressing because it was like, if we don't get it, then if we don't get the visa or the passport, how are we gonna get our child into the country? It's hot. Yeah, it was more, it was a very time-sensitive. I'm looking for the words. Urgent? No, the feeling, like nerve-wracking, I don't know how to, yeah.  You know, we hope this thing pulls through so that we can travel with the baby. And when we went to apply for the Ghanaian passport as well, we thought that because we had passport pictures and they already looked okay, they also wanted to take their own passport picture at the passport's office. But with the Dutch one, you take it somewhere so long as the dimensions match. They are okay with it. So they have the background and dimensions. And the Ghanaian passport, so this is for those who have like Ghanaian partners. Or even if you have a different, if you are also an interracial couple, make sure if you want to get two passports, you know exactly how the other person's process is. So if maybe your country or, for example, her country process took only three days and we literally had to go physically to fill the forms and leave. But with the Ghana one, for example, we filled the forms online, printed them out and went to verify them. So it was a two-step process, went to verify them at the Ghanaian embassy and then also had to take the picture again. And we're not given a finite or definite date that you're gonna pick it up on this day. So do find out what the process is for either of you, if you wanna get a two. If you don't wanna get the two like we did, we pushed ourselves to get the two just cause when we bring him and we are traveling, maybe somewhere or for the next time, we don't want to do it again cause at least it lasts five years for when they're younger. So if you don't wanna do the two, then you know that, okay, fine, you're getting the quickest one and then applying for a visa with the other one if you're gonna be traveling. But still look into the process. Yeah, look into the process. Otherwise the papers are not correct and then you have a problem. Yeah, if it's by birth, they're allowed to have both. I don't know which countries don't allow dual citizenship and passports, but if it's by birth, your child is allowed to have both and it's good to have that advantage. Yeah. So second thing, check with the airline because there are certain baby arrangements that can be arranged. So for example, our airline had like the baby busing net. Busing net. That you can hang on the wall so that you don't have to hold your baby the whole time. So our flight was six hours, which is still doable. But yeah, it was good to know that that's what they can offer and then you had to make a reservation because it was at the special baby aisle where there's a bit more space for your legs as well so you can easily get up and down. So check out the airline, what they can offer and also not if you need to change your baby as the arrow says like, where can I change my baby? Because not every washroom has like a folded table. So we had to ask like, where can we do that? Well, luckily we had very sweet air hostesses. They really liked the baby and... They liked our vibe. They literally came to like, we were just, because you have a baby or a small child, you can go first into the aircraft. So we were just literally sitting down as one of the first people and they came running, checking out the baby and asking how old they see, where are you going, like all these things. And how cute he is. Everybody was like, yeah, he's very cute. Swearing up with how beautiful he looks. Yeah, it was very sweet. And it was also nice because Kami was first behind me because I was in the baby aisle because I had made the reservation for the baby. So he was on my ticket kind of and then Kami was behind me because we weren't able to check in together. But then luckily, I explained to the air hostess, like, great, this is my husband, he's sitting there. And then she's like, okay, we'll try to move you around so you guys are next to each other. So that was really great because then you can just kind of take turns with holding him or feeding him or changing him. So it was nice that we were next to each other, I think, especially for the first flight. So ask if you can or, yeah, ask if it's possible because for us, they moved a few people around and then suddenly they looked around and realized that, okay, not all the seats were filled and then they just moved some people around. We just politely asked and moved some people around. The thing I wanted to draw your attention to for you to remember if you're flying as well is with a bassinet, for example, depending on the flight. So please ask because even when you request for the bassinet and the baby aisle and everything, if you're not the only new mother or only mother traveling on the craft, then it means that you have to be at the airport very early because it's now becomes a first come, first serve thing and as well how young your baby is. So the younger the baby, that will require the bassinet versus maybe an older one that you can maybe handle on your laps the whole flight. So depending on the flight you're using, make sure you have the airport on time. And again, like I said, everything you plan doubles in time. So if you would go an hour before, now you have to do two hours before because you're putting all the possibilities of what if into your planning. Yeah, for your own sanity. Yeah, for your own sanity, which means that if you're gonna go before as well, it means you're gonna spend longer at the airport and you need to pack more as well. So everything is everything in that sense. Yeah, and I think it's also important to check out the airports.  So Schiphol Airport in Amsterdam has a baby lounge. So there are these small pods kind of with curtains around it where that you can close and then you can feed your baby there, change your baby. They have a whole station with running water, warm water, cold water, microwave, even a bathtub if you need to wash your baby if you had an accident or you have a very long travel, I don't know. So it was really nice because it was a quiet part of the airport, you could just really sit down, catch your breath. Grandma was able to get me a coffee. We ate, I think it was a croissant or something. Yeah, we ate croissants. Like just like to wind down a little bit before we went to the gates. And our own snacks that were packed. Yeah, so that was really good. I enjoyed the Rocha. Yeah, and I would also recommend that somebody's bringing you to the airport if possible, like a friend or family because it's quite stressful already to travel with kids. And it's nice that you don't have to think of parking or like how I'm gonna move all the bags or how this, how that. So that's, yeah, and also somebody who picks you up would also be nice. So you don't have to like now request a bolt or a new wear or like hustle with, because yeah, that's too much. You have to arrange both. Yeah, and what I didn't know was at Schiphol, what they do is you can either bring your stroller or your moxie cozy, like the car seat.  And then what they do is when you get to the gates, that's where you give the moxie cozy. To the air hostess or the stroller. Or the stroller. And then when you get out, it will be, it's not in the normal luggage, but in like special luggage. So they put it on the side for you. On the side, yeah.  Yeah, so it's good to also verify that how that works because so our baby was still small. So it was easy to carry him from the aircraft to wherever we were picking up the car seat. But I kind of imagine if you have a bigger child than maybe the stroller, you really need it. So prepare also for that from the aircraft to so you get your stroller or your car seat back. Because if your baby's heavy or something, it might be a bit of a struggle. Yeah. And the third thing, is it third or fourth? Yeah, third or fourth. Yeah. We lost count. We lost count. Also, one thing that I realized after we arrived, dressed for where you were going. Yes. I was wearing the wrong clothes or I wore the wrong attire. And I was like, I mean, we're traveling from a cold place to a warm place.  And then we arrived in Ghana and I was baking. Hot. I was literally like drenched, like the sweat was just like, you know, I was baking underneath. So dress appropriately. If you'd have somebody you go to the airport with and you're wearing your warm clothes, as long as you enter the airport and you can just dress lighter and give it out to the person to go, you know, then do that. And coming to the other airport as well, you have to prepare for a longer, you know, immigration. Yeah. So for us, it took a little bit longer on the airport. So once we landed, we just waited till the whole plane had left because we were like, we're not gonna hustle with the baby in between all these other people. That was the first time I've ever. Yeah. So we waited till everybody was out, then walked out and I know we discussed like, should I make another bottle? And we're like, no, within two hours we'll be out. Huh. We were not out by two hours. Yeah. And what happened was that it was very warm at the airport. Like it was humid. Like a heavy blanket. And of course the baby has not experienced that yet. So what happened? The baby started crying or at least fasting a lot. Yeah. And Carmen was able to utilize the moment. I used my baby. I used my baby. I was like, hey, okay. Since the baby's crying. We're in this queue. I, because we were traveling for a long time and we would also create content every now and then. I always travel with a gear bag, which is usually my carry on most of the time. And it's heavy. And then the baby was also strapped in front of me because Elaine wasn't as strong to have the baby strapped on her on the baby carrier all the time.  And then we're standing in this queue. That wouldn't move at all. For such a long time. And then we go to a certain point. I realized that there's an immigration woman who was doing the directing. Like, you go, you go, there you go. And then the baby started fussing just a little, little, little. And I had the pacifier on. Just took the pacifier out. And I just whispered to him, cry louder. Cry louder. So he's had a crying, like really loud. Because now the pacifier is out. The baby is really uncomfortable.  I was like, yes. But that was fine. Spartylai. Yes. Yeah, so if you have the opportunity, you can do that. You can use it. Yeah, and always make extra food. So we should have made that bottle formula. Always have food on hand. I think it's always good to, unless you're very comfortable breastfeeding in the airplane.  With ours, he slept when we flew away. So we didn't have to do anything. But then at some point, he had to feed, of course. And I tried to breastfeed. But the chairs are very small. And you need some space. At least I need to put your elbow. And I also wanted to cover myself a little. So I had like a scarf. But then it was very hot for him because we're like super close to each other, of course. And then the scarf. So he started fussing. So yeah, it worked half. So it was not enough. So luckily we had like two or three formulas with us. You can always ask for hot water. And it's better to have more ready than to not have it at all on you because we didn't have that when we landed. And then he became very fussy at immigration. It was hot. He was hungry. And then in the end, I had to breastfeed in the back of the car, which was quite challenging because, of course, I also had a long day. And then with our crab potholes and with the bumps. And he latched. But it was difficult to keep him latched because of how the roads are. And it was a whole other thing. So it was too much. So don't try not to stress yourself and have enough food on you. Because that was a big lesson. Like we shouldn't do that next time. If you doubt, just make the bottle.  Always have. You don't know, especially with a newborn. It's better to have a bit more than not enough. So that was a big one. Let me see my notes. You talk about everything? I think so, yeah. Yeah, so that's our pointers for traveling with. And depending on how old your baby is, you may or may not need your headphones, but the flight headphones for babies get one. It's better to have it and not need it. Oh yeah, if they don't like the noise. Yeah, than to need it and not have one with you. So always get it. Luckily for us, when we got into the plane there, the aerosas were like, nah, this plane is not as loud and he's been on six weeks at a time, so he'll be fine. Yeah, and I think what was also nice was one of the aerosas really wanted to make a picture was. So we have a picture. And that's our thumbnail. Of us in the plane, because she's like, yeah, you can show him when he's 18, which I thought was very cute. You see it on YouTube as well. Yes. It's our thumbnail, that's what you've seen. So, but it was nice. It was nice as she thought of that because of course we were not in a headspace to like, oh, let's make memories. But really we're just practical, like, oh, let's get through this flight. But it was nice. And yeah, like ask for help if you can. Yeah, get help. People are generally very helpful because they see you have a small child and they want to support. And yeah, you have people coming up to you as well. Like, oh, like how old? And is that a boy or a girl? And it's a whole other experience in a nice way. Yeah. And also when you see other parents at the airport, you're like, whoa, I see you. You know, you're doing great. Especially those who are traveling with like multiple kids. Yeah, I don't know how to do it. Two even, and yeah, like one is a toddler and one is a baby, and like, whoa. Yeah, and all the stuff and all the entertaining, yeah, you're doing well. So give yourself some grace as well during the travel, especially we were traveling together, but if you were alone, I can imagine it's quite a lot, so. Let me do harder yourself. Yeah. It makes it worse. Yes, and ask for help. I mean, people are generally happy to help. Yeah. Yeah. OK, we hope this was insightful. Enough. And so we'll catch you in the next video. This is me plus you is. Us. And my name is Squammy. His name is Squammy. I'm Elaine. Thank you for watching. Thank you for watching. Bye. Bye.


Rob Music, the chaplain of University of Pikeville, invited a local guest speaker to a weekly event that he hosts in the Christmas Auditorium. I had the chance to speak with Rob and his guest Emma Lowe, who was born and raised in Eastern Kentucky, as she shared her personal story of transitioning from a man to a woman. You can run from your problems. You can drop. You can run from your problems for 30 years. Today is just one of our annual or, I guess, weekly convocation experiences. We have a 45 or 50 minute talk usually by a person in the community or around the world. And what we're really looking at this year is kind of how engaging our world with empathy to make a difference. I've known Emma for a while. Emma actually used to work here at the University in an upward bound program. And Emma last year came and spoke to our medical school about transitioning and medical needs of persons who are transitioning. And our students really gravitated towards her. We have to be honest about some stereotypes about living in Central Appalachia. And I think one of the grossest stereotypes is somehow we're a closed, narrowed group of people. That's not the case at all. I think we're very open, very embracing community. So I think talks like this, to me, aren't abnormal. It's just who we are. I was born and raised here in Pike County, Kentucky. I have a bit of an unusual story as far as what the typical media portrayal of transgender individuals in this country are. So I was a late in life transition. My experiences have been that Eastern Kentucky has not always been the most progressive place. But I've seen a lot of change in that over the most recent years. And I feel like the ideas of diversity help to build a stronger community for us here. But essentially we're just better together than we are separated. I've never felt better about myself and my life since I opened up to myself. Since I admitted to myself who I am and the kind of life I actually want to live. So anyone who can have an opportunity to have that, I say go for it. It beats the heck out of being depressed. It is in the best interest of the community that everyone is in the best state of mind. It's in the best interest for themselves. We do better work. We are able to be of better service to each other when we are accepting of ourselves and accepting of each other. It costs absolutely nothing to be kind.


All right, warm enough in here for everybody. All right, so welcome to our session. My name is Paul Luce. I'm a senior staff engineer at Intel. I've been at Intel for a little over 20 years, primarily doing storage software development. I've worked on everything from raid controllers to Windows drivers to Windows file systems to most recently, this stuff. And this is by far the funnest. So that'll let John introduce himself. My name is John Dickinson. I'm the project technical lead for Swift. And I work at a company called Swiftstack. What we do is we come alongside of Swift clusters and provide monitoring and management tools for that. We've titled this just to talk generally about Swift storage policies. But I really do think that storage policies are the biggest thing to happen to Swift since it was open sourced four and a half years ago as part of OpenStack. And it's really exciting. So with that, yeah, sure. Let's jump into what we're going to do here for the agenda. So John's going to give you a little bit of background information, a little more than what he just gave you. And then a Swift primer, just to get everybody kind of up to speed on the super basics. There's a lot of other Swift sessions later on in the week. So we'll encourage you to attend those for more details. And then I'll jump in and take over and talk a little bit about why we thought storage policies was a good idea. What storage policies are. And then probably most importantly for everybody is how do you use storage policies? What new usage models do we enable? And then we'll wrap up with some status and we'll wrap up the project. So let's give a little bit of background to start with. Swift is really important, I think. And it's important because fundamentally I think that everybody has data. It's always growing. And you need to have ownership of everything of your data. And what I mean by ownership is you need to be able to know what touches your data from the hardware to the software that's managing the storage to actually the tool chain that's doing that. And the only way you can actually do that effectively is with open systems. It doesn't necessarily mean it's running inside of your own data center, but it does mean that you can get involved in the community, that you can influence the source code, you can see that, you can edit it, you can contribute back, you can affect the direction of the project, you can get involved with that. That is why I think that OpenStack is important. And that's why I think that Swift is important as a storage system here. But there's a lot of practical concerns you have as well with a storage system. It's important that we actually can store stuff and do things well. And so we need to actually offer something fundamentally new besides just the fact that you can see the source code and vote in an election sometime. So what is that? Why are we here as far as Swift? What does it fundamentally offer that's really valuable? I'm going to go into a lot more detail tomorrow. And I'll tell you when this is a little later at Swift 101 talk, we'll go into a lot of example use cases and architecture stuff. But to start with, the high level reality is that the old paradigm is if you need to store something, you plug in a hard drive. And then when you fill up that hard drive, you plug in another hard drive. And then you're stuck with this hard problem of siloing across these different hard drives and remembering how all that kind of stuff works and where you stored the things, it's really painful. And then you add in the complexity of the reality that hard drives fail. And so you buy a raid card. And basically, you've just made your problem bigger because instead of just having individual hard drives, you've got these individual silos of giant raid volumes, which have their own problems with rebuild times and performance and these sort of stuff. So the really important mindset to change, I think, is the fact that you have a storage system that abstracts this. It separates the data from the storage media itself. Then you can fundamentally do new things. And some of those new things actually involve supporting the existing modern applications that people are building today. So people are building things that demand scalability. People are building things that demand global access. Just to take a little brief note, I heard a new use case just yesterday that was really awesome. It shows a great use case of Weiswift is really important. And this kind of feeds into where we get into the flexibility of storage policies. You may have heard there was an airline that went missing in the Indian Ocean. And in order to find where should we expect to look for debris and stuff like this, they found a website that was hosted in Australia that was, you click on any point in the ocean, and it just kind of shows where the debris field is going to be in one, five, 10 years from now, which is really kind of a cool thing. It's fun to just click around to play with it. Well, what happens is apparently the Guardian got ahold of the story and just destroyed their website. And they were just a standard normal lamp stack sort of thing. And they figured, well, we could just add more Nginx servers and scale it that way. But it turns out they were running on the Nectar Research Cloud that the Australian government has set up. And their data was stored in Swift anyway. So instead of this figure how to scale out our web servers, why don't we just point the clients straight to Swift?  And boom, their scaling problems were gone. The storage engine itself was able to offload those hard problems of storage from the application. That's why Swift is important. That's where Swift fits in. And I think that's one thing that was just really exciting to hear about yesterday, sitting in the hotel lobby. So to kind of back up and get a little, finish up the summary here, talk about how the important thing is to abstract away storage volumes from the actual data. And you can do that. You look at the evolution of storage media and technology there. Storage media fails. You want to deal with heterogeneous stuff. Old things go away. New formats come along. You may still have old iOmega zip drives somewhere in a drawer. But you can't get any data off of them because you don't have a zip drive anymore. And if you did, you don't even have the right plug on the back of your computer to plug it in anymore. So the point is new technologies are coming along. Optical stuff, memristors, fancy ways to write onto hard drives. And we need to be able to abstract that behind the storage system to take over that hard part of figuring out how to effectively store it from the application. So where does that give us? We can do that. But we still need to be able to support these modern applications, say, for example, hypothetically speaking, a website that gets mentioned by a newspaper about a major news story, or mobile applications, or user generated content of all this kind of thing. We've got to be able to support this massive concurrency and still remain highly available as the system operates. So that's what we do with Swift. That's where we are. So let's talk just briefly about how it works. And then we'll build in where storage policies come in and talk about the flexibility and the power that that gives you to directly match your infrastructure to your use case. What is the Swift API? We talk about it having a REST API. And you hear about accounts. And there's an object storage system. But how does that actually work? This is what a Swift API request URL looks like. You've basically got a few key parts. The most important pieces there are the account, the container, and the object. That's the things that Swift stores. And the account keeps a list of containers inside of it. Containers keep a list of objects inside of it. Both of those keep a little bit of aggregated metadata. And then the objects where your data is actually stored. So when you make a request, let's write a new object. Well, you send an HTTP put to your object name. And you get back an HTTP response code. And when you want to get the object, you send a get. And you get back 200. Here's your data. And that's the API. And there's a lot of extra features and functionality we've added onto this. But this is the important piece as far as how do you read and write your data? What are the parts of Swift? How does this work? So the client, the user who's needing to talk to the data, remember what I said, there's this system that's distracting away the storage volumes. The user talks to the proxy server. The Swift has basically two pieces, a proxy server and a storage server. The proxy server is what implements most of the API, but importantly takes all of those API requests and then coordinates the communication with all of the storage servers. The storage servers are responsible for talking to the actual physical media. What this allows you to do is a couple of really cool things. First, at the lower level, is that the storage servers then can deal with hard drive failures, deal with different sized medias, deal with different types of storage volumes and things like that, without ever having to worry about how that propagates up to the user. The proxy server, on the other hand, is able to know when servers go up and come down, when you need to deal with making sure your data is durably stored in the system and coordinating all of the responses there. And what this allows you to do is survive drive failures, server failures, network issues, potentially even rack failures, DC failures, and still have a highly available durable system. So, quick question here. Does the client ever talk to a hard drive when using Swift? No, it does not. Does the proxy server really have to worry about talking directly to hard drives? No, do we really care about what kind of hard drives plugged in and they're all the same size or they're all the same manufacturer model number? This gives you a lot of flexibility as far as how you are able to build out and scale your system and keeping very high uptime. So, here's the big picture. We need to know, we've got these nice logical block diagrams, very simple, but what does it actually do? What does it look like in production? Well, a lot of times, you're going to have building out with some sort of load balancing tier, talking to proxy servers, and then the proxy servers are going to have basically what you've got here is your capacity tier that has different types of storage nodes where the hard drives are. You've got auth servers that are associated with that, being able to plug in different auth systems to integrate with your existing environment. So, when an object is uploaded, the request comes through the load balancer to a particular proxy server, and then that data is written out three times, because Swift is a replicated system. Once it has been written out durably to those drives, then the storage nodes can send a successful response back to the proxy server. The proxy server correlates those responses, make sure that we got at least a quorum written out to all of those hard drives, and then can respond with a success back to the client. Now, the converse is that when a request comes in for a git, the request comes from the client, talks to one of the proxy servers. The proxy servers look up one of those locations and is able to get the data from one of those copies, and then it's sent back to the client. So the important piece that comes in there is what about the data placements? This is an overview of how we're actually assigning. How do you choose where a particular object lives on those hard drives? You've got a lot of hard drives. You've got, you know, you may have dozens. You may have dozens of thousands of hard drives plugged into one cluster. And these are hard problems to solve as far as effectively balancing this. The data placements is done based on a server, a drive, a zone, and a region. And I think I have it better. This is what I wanted to show. So it's kind of a tiered system here. So what happens is first we want to make sure that we put the data as uniquely as possible across all the drives. So if you have more than three drives, make sure that every copy is stored on an individual drive so that one hard drive failure does not cause any sort of durability or availability issue. And then you propagate that up with servers, zones, and regions. And so when a request is made to create a new object, we calculate out where that should go and choose three of those. So in this case, we're going to choose a hard drive on three unique servers. Notice we only have two zones. Zones oftentimes deployed as an individual rack. And therefore, you can lose even an entire rack and still have access to your data. So how do we do this? Let's get probably the most technical piece here. Paul's going to actually go into a little more detail in a bit about this. But the way we do this is with a consistent hashing algorithm. Paul's going to go into detail in just a bit about that. But to give you a brief, high-level introduction to hashing, we all understand it because we've all used it for all of our lives. So think of an encyclopedia. Remember when you were a kid, you had to go home and look something up in encyclopedia before Wikipedia. And you needed to go do your book reports on the jungle, so you, the region, not the book.  And then you want to go look at kangaroos. So you pull out K. And then you do ostriches.  That's a hashing algorithm. You've taken a broad subset of all the topics, and you've figured out a way to fairly evenly divide it. And what's really great about that is you can kind of further subdivide and very quickly go look up where your data is. So keep that in mind when Paul goes into kind of a little more about how the storage policies are implemented. All right, thanks, John. OK, so why storage policies? Well, as John just took you through, and we'll go through further in detail later this week, Swift is pretty successful today. It's pretty flexible today. But as with any software project, any really project in general, there's always technical opportunities. So we've identified a few technical opportunities and have answered them with this thing we're calling storage policies, which by the way has been a great collaborative effort between multiple companies in the community. It's going really, really well. We started working on this, I think, last summer, right, John, a lot of great contributions. I want to say thanks to everybody who's pitched in on this. But some of those technical opportunities, some may be obvious, some may be not. As you may or may not know today, Swift is a replication system. And your choices are how many copies? So you get it one way or the other for the entire cluster, whether that meets your needs or not. And it does meet a lot of people's needs today. But we think we can do better. How about all our old nodes equal? Well, most clusters deployed unless they were just brand new purchased are not homogeneous. You've got some machines that are faster than others. Maybe you've got some hard drives in some systems that are SSDs. You've got some legacy stuff. You've got this mattering of stuff out there. And as John said, although you don't need to care which hard drive is in which system, sometimes you might want to care. And one example with today's Swift is the use of account container databases on SSDs. But today, you can't use those SSDs for objects, just for your databases. So what if you wanted to use them for objects? And what about adding something like erasure codes? So I think everybody's probably familiar with that. If not, we'll talk a little bit about it Wednesday afternoon in an Intel session. And you'll be hearing a lot more about it if you come to Paris, hopefully. Today, it would be very difficult to bolt something like erasure codes into a replicated framework. But with storage policies, we're really building in the extensibility to do things like erasure codes. We'll see more about that later as well. So with storage policies, as John alluded to, we're also, besides bringing some really interesting technical ideas, we're really increasing the modularity and extensibility of an already successful system. So this is a pretty cool diagram that John had put together that gives you a couple of different perspectives on where the knobs are that you can tune with storage policies and with some other features that were recently added to Swift, like disk file, if you're familiar with that. And I think Pete has a design session this week on disk file architecture. So that'd be a good one to check out, too, if you're on the design side. But what this picture shows us is, looking at the top, the client now has a selection of things as well as the operator. And from two different perspectives, there's different things that you can tune. You can choose your hardware, which is primarily a deployer thing, but your client has a say in that as well. You can choose your encoding, like I just talked about. Policies will allow something like a ratio code to come in the picture. And by the way, it is coming into the picture right after policies. And then your protocol as well. So this is your back-end disk protocol. So when you combine things like storage policies and disk file, you really get an incredibly modular and extensible system. Okay, so can't really do the topic of the Swift ring justice in one slide, but we're gonna go into the hood just a little bit because the concept of Swift's consistent hashing ring is really a fundamental part of storage policies. It is the foundation of where storage policies really gets its enablement from. So the Swift ring is actually implemented in the code as a static data structure. So I'm not sure how many people are super familiar with this, but again, there's a whole lot of really good resources. John and Sam at Swiftstack have a video out there that's really good. Greg Holt has a series of articles on consistent hashing ring that's really good for background technical information. But actually in the Swift implementation, what we'll be talking about here is this conceptual ring and these idea of partitions, which are sections of the namespace within the ring, not to be confused with disk partitions, which is probably a pretty common thing for people new to Swift. I know it was confusing for me. But the way this works is you take a URL like what John showed earlier with the sample of the put and the get, and we calculate an MD5 of that. You can see that up at the top. And that provides us an index into the ring that identifies a particular partition for us. And each partition then maps to a set of tables that are part of this ring structure. So this ring structure is not actually a circular structure of some kind. It's two tables and a few other elements with some descriptive information about the ring. But what happens then, I'm not gonna talk about the names of these tables because they're hard to read for a human. But let's say this thing works, okay. So let's say we did that lookup of a URL. We'll take that index and we'll go to the first table and you can see we'll go to the first table and that will pull us a row in the first table. From that row, depending on how many replicas we have, we have a certain number of columns. In this example we've got three columns because we have a three replica ring. So that row gives us three other indices that we use to go to the second table. And that second table is actually a list of devices. It's a list of nodes, right? Or more specifically, it's a list of hard drives, right? With their IP address of what storage node they live in. So this static structure is responsible for the placement of all of the data within the cluster, right? And if you can imagine when I mentioned earlier the account and container databases can live on SSDs, what gives them that capability is they have their own rings, right? They have their own set of tables. When you go to make an update to a container database, it doesn't go to the objects ring, right? Or the objects set of tables, because that would point it to a bunch of hard drives. It goes to a special ring that you create for the container databases that the cluster administrator has specifically put SSDs on to help eliminate that as a bottleneck in the system. Okay, so that's just kind of a quick intro into the ring. We'll go backwards and forwards a little bit. Okay, so what are storage policies? Well, as you might guess based on what I just explained, hopefully you might guess, it's the use of multiple object rings, right? That's really the simplest explanation. And honestly, if you're super familiar with Swift, that's probably all you need to say about it. And you go, oh, I get it now. Boy, I can do a lot of stuff with that. Simple little concept. So here's an example. Three different rings, three different policies. So here's the first one, trip replication. As we saw in the last couple of slides with triple replication, you get three locations from your ring when you go and say, where do I store this file? Or where do I get this file? With storage policies, you can do something like reduced replication. Okay, so only specify two locations. So you might say, well, Paul, earlier you said, I could pick any replication level I wanted, two or three. So what's the difference with storage policies? The difference is this can be in the same cluster. You can have two X and three X replication coexist in your same cluster. And now you give that choice to your application. So it can decide what level of replication it wants. Okay, and then last example here, coming soon to a theater near you, erasure codes. Okay, so with the erasure codes, we'll use the ring to store the location of all of our data and parity information. So you'll get a bunch of locations back from the ring that's assigned to the erasure code policy. Yeah, this thing's not working. Okay, so the second part of storage policies, the second sort of big feature, I should say this is a fairly significant undertaking. Not only as far as like what John mentioned, it's importance to the project, but the sheer size of the effort, we've started last summer. And I think by the time we merged to master here soon, I've been saying soon for weeks, but it really is soon. We'll be close to 5,000 lines of code out of a total of including test code about 60,000 lines of code. So it's a pretty significant change, touches a lot of places, which is kind of amazing because it's so simple. But the second big piece of it is the use of a new container tag called X storage policy. So this is the actual, the only change to the API, which is the other really cool thing. All this capability, all this feature, one little metadata tag does it all. So when the application creates a container, it just provides this little tag and it gives it a storage policy. Says, okay, everything, when you write an object into this container, make it erasure coded or make this container a triple replication, make that container a double replication. Can do it all within the same cluster. And then your application just picks its durability policy based on which container it writes to. Or geographic dispersion. It could be one global Swift cluster and you have an EU set of nodes in US. Yeah, we've got some usage models coming up that'll talk to what John was just mentioning. But yeah, it still floors me. It's such a simple concept with so many different usage models that we enable with this. Okay, so I'm gonna flip through the animation on this one. I won't go through this in detail, but I did wanna give everyone here a sense of sort of the size complexity. I don't really get complexity by looking at this picture, but this is your super high level software architecture diagram of Swift, showing the components that run on the proxy server at the top and down at the bottom the components that run on the various servers, container, object and storage. And the ones that are colored in orange are the ones where we've had to do a decent amount of work to get storage policies to become a reality. So just to give you a feel for it, and if anybody wants to talk any more of the details about this, of course, we'll be around all week. Okay, so let's talk about usage models. I sound like a lounge singer. Be here all week. So I think some of the fun things to really drive it home to say that, okay, we've got this level of work. This is kind of why we're doing it. Where Swift fits today, how it's gonna be extensible.  So we know how it works. So I think the first obvious use case would be, I'm gonna store web content, but I need to store thumbnail images. So I don't wanna stand up a second Swift cluster just to store thumbnail images. I can recreate those any time they're missing, but it sure would be nice if I could store them and didn't have to recreate them on demand for every request. Great, let's create a storage policy that has thumbnails. And let's put our source images in the normal triple replicated for high durability storage. That way I know that I can always fall back to my durable storage, recreate the thumbnails, put those into the thumbnail container, and everything's gonna continue to scale very well. And I'm gonna save a lot of money on hard drives because I don't have to have that third copy for the data that I know I can recreate. It's not as important to me if I lose that data. I don't want to, but that data I can recreate. Another usage model, performance tier, right? So I mentioned this when I talked about today's usage of SSDs for account and container.  Now when your application knows that it's really important data needs lower latency, you can now store it to an SSD policy and it doesn't have to be three copies. It could be two copies, could be one copy, it could be whatever you want. Again, because it's its own ring. And I think something important to note on all of these things is that notice that in the iconography here, these drives are actually on the same physical hardware. Some of the drives are on different physical servers, some of them are on the same. It still means that these things can overlap. They don't have to be completely discreet to rings. It means that you can, again, very much tailor it to your use case. So the last one I think is very interesting is looking at the basically the geopolitical realities of where we are, especially outside of the United States. It's very nice just for locality to have different regionalities. Wouldn't it be kind of cool if you could just say, I've got an East Coast location and a West Coast location. But especially outside of the US, for example, in a lot of the EU and in Australia, they have laws that say you cannot let this data leave our border. So it would be really nice if you could say, I've got a global Swift cluster because I need to have locality of access for some subset of data, but these backups are this personally identifiable information. I need to guarantee that it doesn't, for example, leave Germany. So I'm gonna put it in my German data center. I'm gonna put that storage policy there that says this data is only gonna be right there. And that's I think one of the very nice things, both for I need to constrain my dispersion, but also to say that I can expand my dispersion if necessary in order to have locality of access. So that you can, this is kind of a follow on of the global clusters concept that Swift supports today. But now you can say, I wanna have this data that's going to be globally dispersed. This one's gonna be East Coast. This one's gonna be Asia. And you can all manage that in the same Swift cluster. And yeah, and last, but definitely not last, last in our slides is I'm certain when we get this out there, people are gonna come up with usage models that we didn't think about as developers. These are things that we've thought about sort of, in a vacuum, but as soon as this gets out there, I think we'll see a lot of other stuff take off. But erasure codes, as I mentioned earlier, there's your container with triple replication. We had erasure codes and now it's again, policy-based via container. So an application decides that it doesn't want triple replication and needs to save not only disk space, but power is important as well, right? If we're talking significant number of hard drives, power in the data center, cooling, all that stuff comes into play, all that capex, op-ex kind of stuff. But you can see here now we store EC fragments instead of objects. And again, we've got another talk on Wednesday. We'll go into a little more detail on erasure code and stay tuned, because we're working on it hard over the summer. So where are we going on this? If you keep up with the OpenStack development mailing list, you know that I've been talking about this for a little while within the community. In fact, we were talking about it to some extent in Hong Kong. And as Paul said, it's been a massive effort from a lot of people. And we're not gonna release it before it's ready, which means that basically another way to say that it's your data's important, we're not gonna throw some feature over the wall just to have a feature by a time-dased release without having proper documentation, proper QA, proper vetting, proper design, all of that sort of thing. So for those reasons, it was not a mistake, but it was something that was, we did not feel comfortable by releasing by the Ice House release. Since that, which was really just a matter of a few weeks ago, we've been working quite a bit on getting those last little feature gaps covered, making sure we can do the appropriate documentation and QA and things like that. Storage policies, I'm really hoping should be starting to hit master or the patch proposals to master very, very soon no matter if I would hope this week or next week, we'll see. And then it should be continued to go through the integration process with full developer review and QA vetting. And so at that point, we should have that in the next Swift release. Swift release is a little more often than the six month open-stock cycle. And we will definitely have this in the Juno release as far as the combined open-stock release. So we're looking forward to that continuing and being finalized real soon now. After we're done with the storage policies, we have that framework built out. We'll be building out the erasure code storage policy so you can manage that effectively. And that's what we have on this. I think we have, let's see, a about 10 minutes, nine minutes. And for any questions you may have, we've got a microphone in the middle. But a few things I wanted to call to your minds later this week, that we have quite a few sessions that are around Swift and Swift sort of things. Everything from I'm a developer, I want to extend Swift, how do I do that? I've got some use case, some case study sort of things of real world users who are deploying it and a lot of great stuff. Couple things I wanted to point out. We've got a workshop on Thursday where you can walk away with a Swift running on your own, your own personal Swift cluster. It's a great gentle introduction to I want to deploy Swift and play with it. And a couple more that are not on this slide are at 340 today in this room. Joe Arnold's going to be talking about global clusters. And then at in B101 tomorrow at 11.15 in the morning, I'll be talking with Joe on a Swift 101, just kind of here's the technology and architecture for beginners. It's very gentle introduction to what Swift is, how it fits, where it works. And with that, I think we have a couple of minutes for questions. And we actually also have a Wednesday session at around three, I don't have the exact details, it's called Intel Contributions in Object Storage. So come here a little bit more about policies of Racial Code and Cosbench also. So question here? The question is, what's the granularity of the storage policies? Where are they specified? Is it the object, the container, or the account? And the storage policy is determined, is set at account creation. And so you can create, for example, a Racial Code container. You can create your global cluster container, your EU container. Container creation, you meant to say. Yeah, create container creation, sorry. Yes? Great question. So the policy sounds similar to tiered storage. So the question is, can you change the policy after the fact? And very explicitly at this point, we are keeping that outside of the scope of what we're doing. So furthering some of the design principles that we've embraced within Swift, we are ensuring that we have all of the tools and hooks and power available to be that good storage engine. And if you need to have a simple script that can migrate from one storage policy to another based on a certain set of rules that is entirely possible and supported and pretty simple to do, it is not something that's going to be included inside of Swift Core's code today. Yeah, so the question is, is there gonna be APIs for this? To start with, what we're doing right now, what is in scope today, is the ability to, the API is simply move it from one container to another. And you can use that using Swift server site copy feature to say copy this data from that container to that container. And there you go. Over here? Yes, exactly. So talking about geotagging and saying that what does that mean and how is that done? And it's defined in the ring and it was just confirmation that yes, what you would do is if you had say a New York location that you would, the drives that are physically located in the New York region, you would add to the New York ring. And that's how that would be done with geotagging. Let me add a little bit to that, John, just to clarify. That's an example of a usage model that isn't completely fully baked yet, right? That's something that as a complete solution, we'd probably wanna file a blueprint this summer and do a little bit more looking into the account and container side because those pieces aren't covered by storage policies. So if you had a compliance regulation that required the metadata to also be in that geo, we need to think about those kinds of things. Also at Intel, the software and solutions group is working on a similar feature for compute networking side and we'll wanna look at ways to tie those together so that you could actually spin a VM and have all of your storage be specified and isolated to the specific geotagging policy that's associated with the VM. So there's a lot of work to do still there. That was just sort of an example of what we can do now that we couldn't do before. In the back. I'm talking about the life cycle management of the data. Can you nest policies? So yes, to a certain extent. So let me, there's two questions. The first one is the automatic tiering and basically some sort of HSM sort of thing. No, that is not going to be just like the question up here. And that's not something that's in scope for SWIFT right now. The second question about nesting policies. I wouldn't use the word nesting, but yes, what you're looking for is entirely possible to say that this is gonna be erasure coded in this particular geography as opposed to this is gonna be replicated in this particular global state, something like that. Yes. Can a single disk drive host multiple containers with multiple policies? Yes. And in fact, that's what happens with SWIFT today. There are the data on a particular hard drive is data from all over the cluster. There's not, in SWIFT, the API is not such that you are going to be provisioned a pool of storage and then it's assigned to a particular thing. It is really you are given a namespace which is placed across this given set of hardware. And one other relevant point to that, we've designed storage policies such that the same hard drives can be used for multiple policies. So although our pictures for simplicity kind of showed these policies mapping to these drives, you'd have one set of drives that support three, four different policies and not have to worry about separating them. You can make them separate if you want, but you don't have to. Yes. No. No, that is not the case. Well, you do not divide the drive up when you're doing this. So what happens is if you have ever looked at, to get into a little of the technical details, if you look at how data actually is laid out on the drive on the local file system, for example, your object data is in a high level directory called objects. So we're going to have a sibling directory for that for each additionally defined storage policy that has data on that drive. So if you have your tuna fish storage policy, as opposed to your cupcake storage policy, then maybe tuna fish's storage policy index one will have objects dash one as a high level directory. So you're not carving up a single hard drive into multiple file system partitions and storage volumes. What happens is that you have one storage partition. Basically you can think of this as one failure domain. This is the hard drive itself, which is storing the data on top of it. And Swift is managing the global state of it. That particular failure domain doesn't have a global state. Across the cluster. So I'm kind of thinking whether these policies where you can use one drive to want to hurt that. So the talk about data placement and making sure that things are evenly balanced across the drives and does this hurt that because you might have one drive that has more policies on it than another. Yes, you can configure a policy that says, you can put one drive on all of your policies and overload that drive. But in general, this is not affecting the data placement. But as an operator, yes, you need to manage your sizing of drives. I think we have time for one more question in the back. So if I understand you correctly, the question is if I have an existing triple replicated policy that's globally distributed and I add on another reduced redundancy policy of two replicas, does that affect the data placement of the first policy? And the answer is no. Each of the policies because they are implemented as individual rings are actually fully autonomous and at impotent. So a change to one does not affect the data placement of another one. With the caveat that if you have one small hard drive that has, you know, every policy is gonna use this one hard drive. Yeah, that one might fill up more than one. It's only used for one policy. But that's more of a operational management system, not something as far as the general data placement. So I hope that you have a better understanding of how storage policies work and why they're awesome and why I truly think that they're the biggest thing that happens to Swift since the whole project was open sourced. It's really exciting. I want to say a particular thank you, especially to Paul for helping me today and also doing the lion's share of the work on the coding of the storage policies. Also Sam and Clay at Swiftstack and several people at Red Hat have been very instrumental in making sure that this is going to reach the light of day. So thank you for your time. We'll be here all week. Come to lots of the other sessions and have a good day. Thanks everybody. Thank you.


Financers of Wagner-Messnery chief, Pregersen Kertrin Femme, are said to be investigated. Russian President Vladimir Putin said today as the Wagner group leader arrived in Belarus. Speaking to soldiers from the Russian army at a meeting in the Kremlin, Putin said, it always respected Wagner's fighters, but that the fact was the group had fully financed from the state's budget. He alleged that he received eight or six bail-in rubbles from the Defence Ministry between May 2022 and May 2023. Belarusian President Alexander Lukashenko confirmed today that Pregersen has arrived in Belarus following a short-lived air-murter in Russia. Pregersen's exile had previously been announced by Kremlin with Lukashenko ardent that the group will be welcomed to stay in the country for some time. To hit the notification button, so you get notified about fresh news updates.


Well, welcome everybody. To be honest, I didn't expect such a crowd, but I'm happy you're here. So this is everything you ever wanted to know about OpenStack at scale. Seriously, everything. No, but we're going to go through our views on OpenStack scalability and some basic scaling models and paradigms. And let's see here. Maybe I was supposed to explain this in the Y side. OK, right. So we're going to talk about basic scaling models. We're going to talk about how they apply to OpenStack, where they fail, where they succeed. But I just want to introduce you to the motivation of this talk. So I work with Merantis full-time. Work with OpenStack for the last four years, Merantis for the last two. And I think I have heard this question more than any other question about OpenStack. How does it scale? How do you go bigger? I think the other question is, how do you upgrade it? So we want to do a general overview, component by component. And then we will have some time for questions at the end. Let's see here. Here we go. So there's kind of our overview. Three core sections, scaling models, core OpenStack services. And then we have some special buggers here at the end that we'll make special mention of. I'm going to go ahead and turn it over to Randy. Hi, everyone. My name is Randy DeFow. I've worked in Merantis for about a year now. And my background is a little less on the system architecture side. So Mike specialized in that. He's run very large OpenStack Clouds. I come a little bit more of application and cloud solution side, but I have worked with a lot of different types of distributed systems over my career, including some of the big data systems and distributed consensus algorithms. So I found some of the scaling problems in OpenStack to be pretty interesting from that perspective. So before we jump into the component by component, I just wanted to cover some general scaling models and paradigms, which I think help frame the problem. I mean, one thing that's really interesting to me about OpenStack versus systems I worked with in the past like Hadoop, is that OpenStack is a very complicated system. And there are so many different components that scale in different ways. And so one of the big challenges, I think, for those of you who maybe have worked with other systems in the past and are now looking at scaling OpenStack, is that you have to be really specific about which particular bottleneck you're trying to address. Because with OpenStack, what I found personally is that if you dive into a problem, you can quickly become confused about looking at a lot of different possibilities. And so you have to be really careful about identifying the actual specific component that's the bottleneck before you figure out the best way to solve it. And I can say that from somewhat painful experience, even over the last couple of weeks, as Mike knows. So anyway, of course, one scale-up model that's traditional one is scaling up, just throwing more resources at the problem.  Michael talked a little bit about how this works, even for the MySQL or Postgres database that's seized by OpenStack. And this, of course, works until you hit some limit that is just too hard to overcome. And what's interesting is that you can think about this limit for a specific thing, like MySQL database, so you can only throw so much RAM at the problem, or you can only increase drive speed to a certain extent. But if you look at it as a system as a whole, so if you start thinking about a particular component of OpenStack, say, like a Galera cluster, even though it is, in one sense, a distributed system, there's still this fundamental bottleneck you hit if you look at it as a system as a whole. And so the example I always fall back on is distributed consensus algorithms like Paxos. You reach a certain size, the quorum gets too big, and adding more nodes doesn't help you any because the overhead of the voting gets so heavy. And we actually see that in OpenStack. For instance, one customer I worked with tried to have a Keystone database that was distributed over many sites. Each one had a four node Galera cluster. And talking about going to six sites, a 24 node Galera cluster, if you think of that as a single unit, it just starts to get too big. So trying to scale Galera beyond 15 nodes can be really difficult. So of course, the next general scaling model is the scale out model. I always love the Grace Hopper quote because I just have this picture in my head of a gigantic ox for some reason. But of course, we can try to, a lot of things in OpenStack, you can just try to add more pieces, more nodes to the problem. So a lot of the OpenStack control services you can scale out by say, instead of running three Keystone controllers, you can run five if Keystone is a bottleneck for you. But of course, again, looking at it as a system, the key thing you have to be aware of is there's always a tax you have to pay. There's some level of coordination that you have to invest in to make those five Keystone instances work together as a coherent unit. And in OpenStack, that's really important because those overheads tend to become some key bottlenecks. So things like RabbitMQ, which is the glue that kind of holds a lot of these pieces together. And Michael talked more about that later. So again, just in general, when you're looking at scaling problems in OpenStack, it's really important to think about the difference between looking at components and looking at systems. Because by scaling out a particular component of OpenStack, you're actually scaling up the system as a whole, and you'll end up running into some of these bottlenecks like I just talked about. For instance, those of you who've worked in big data before, you know that Hadoop, of course, as a distributed file system, can scale out to be fairly large. But even in Hadoop, you end up with things like the name node, or in some cases, even the resource manager in Yarn, becoming a new bottleneck you have to worry about. And so in the Hadoop community, the name node, if you're not familiar with it, is the part of the Hadoop distributed file system that knows where different chunks of files are on different actual servers in the distributed file system. And this started out just being one of those, and so it was definitely a scaling bottleneck. So if you just kept adding more data nodes into your Hadoop cluster, it could be a real scaling issue. And so the Hadoop community has started to look at things like sharding that, dividing the distributed file system into different namespaces. And you'll see that same pattern happen with some parts of OpenStack. So for instance, even if you look at the data plane in OpenStack, you can keep adding more and more compute nodes, but at some point it starts to stress out things like the Nova scheduler. And so the OpenStack community is starting to look at things like federation and sharding. If you've ever heard of Nova cells, which is a somewhat still an experimental feature, but is gonna become a lot more prominent in Mataka and Newton, there's actually really good talk on that a couple of days ago, you can look up the video on it. You'll see some of these same patterns being repeated now in OpenStack. So just wanted to focus on one particular example that I think is kind of interesting because it kind of ties into some of the buzz you'll hear about containers and how they may fit into the OpenStack picture. So if you look at schedulers, and I'm personally invested in this because I've been helping someone kind of troubleshoot a potential scheduler bottleneck the past few weeks, the Nova scheduler to me, even though if you look at it just in isolation, it does have a scale-out model. You can add more Nova scheduler services running on more controller nodes. But from a system point of view, it's very much a centralized system. The Nova scheduler has to have global knowledge over the capabilities of the entire data plane unless you are one of the early adopters of things like Nova cells. And not only that, but it has to, through the use of filters, it has to have a lot of knowledge about what tenants may be asking of it. So tenants can pass in hints like I would need a flavor that's tied into specific host aggregate properties. Maybe I need to know that it has S or IOV capabilities. But that scheduler, because it has to have this global view of pretty much everything about the system and everything tenants may be asking of it, the Nova scheduler ends up being a common scaling bottleneck. And what we've seen in practice in our scale lab is that you can end up running the scheduler up to about 12,000 VMs on a couple hundred of compute nodes. And depending on your configuration, your mileage may vary. But there's a ballpark figure where we think the scheduler monolithically is gonna become a real bottleneck. And it's interesting to compare that to some of the newer container-based schedulers. Apache Mesos is one I like to look at has a much different federated scheduling model. So there's a separation of concerns there where the scheduler, Meso scheduler itself really doesn't want to know everything about the entire state of the cluster and everything tenants may be asking of it. It really just passes on resource offers from different parts of the cluster and then application frameworks can choose to accept or reject those resource offers. So in some ways, it's a lot more lightweight of a scheduling model. And I'm kind of oversimplifying, but it is a useful comparison. What's really interesting about Mesos is there's production systems that are known to run at 50,000 nodes, which to me is several times larger than what we would feel comfortable with for a single monolithic OpenStack data plane. And this is really interesting because some of the new container concepts floating around OpenStack like Magnum actually introduced this kind of separation of scheduling concerns. Magnum, it's a fairly new project. I don't have a lot of extensive field experience with it, but projects like Magnum or the Kubernetes packages on Merano, they'll actually use the Novus scheduler to carve off a chunk of your OpenStack cloud directly or indirectly orchestrated via heat or something like that. But they'll take a chunk of Nova and other resources and just make it available to one of these other schedulers. And then you can go and invoke cube control if you're using Kubernetes or you can invoke marathon if you're using Mesos and use a container orchestration engine. So you kind of get the separation of scheduling concerns. That's an interesting model. I think it's kind of an open question operationally, how well that works. There's some other questions it raises. But anyway, these are some of the newer directions that you may see in OpenStack over the next few releases. So enough of the theoretical, let's get down to I guess some of the nitty gritty about different OpenStack services. So a lot of this information is summarizing some field knowledge that folks like Mike and some of the other solution and system architects at Mirantis have put together over the years. We'll start with Keystone. Keystone, of course, is a central service for OpenStack. A lot of, pretty much every other service has to interact with Keystone to some extent. Keystone, to me, is a pretty interesting system because the Keystone API services are really easy to scale. They're active-active. Behind a load balancer, you can scale out, just add more Keystone controller nodes into the control plane. But what's more interesting about Keystone is the authentication and authorization backends. And here, depending on your back end, you have a lot of different ways you can possibly scale up or scale out. So for instance, if you're using internal Keystone, just have it in the same MySQL database the rest of the control plane uses, then it can scale out the same as your MySQL database. If you're using LDAP, I really think of it as a scale-up model because LDAP is typically owned by some other department and I make it their problem. But LDAP is a system that has proven itself can run at pretty large scale. And then, for some of the newer deployments, particularly if you're running a multi-region, you can actually have federated Keystone. So you can have local Keystones that actually defer back to a centralized Keystone for authentication. So a lot of different ways you can try to shoot to scale Keystone, depending really on your deployment model. And with that, I'll turn it over to Mike and he's gonna walk through some of the other services. Thanks, Randy. So Nova is an interesting case. I think pretty much every open-stack deployment out there probably uses Nova as one of the core services. Nova, we've talked a little bit about the scheduler. Nova has been deployed at scale, at different locations. We have some examples, Acorn, Bluehost, CERN, Rackspace, and primarily because it has this idea of application level sharding, we'll call it Cells, right? So Cells V1 is not for everyone and not recommended unless you have a team of experts to help you out. But Cells V2, which is gonna be implemented in Newton, it will become the way that Nova works. You will always have a federated model that you can fall back on. So real good progress in Nova. Very quickly, I'm gonna go over the components. This is an intermediate-level presentation, so I'm not gonna go into great depth, but we'll go over them, we'll talk about their roles and how they scale. So the API service in Nova is a pretty typical open-stack service. You'll see an API for almost all open-stack services, and they all kind of look like this. They're launched via Apache or Nginx, ModWizki, or you can use the Wizki launcher directly. But you just kind of spin up more workers of that.  It's pretty easy. Scheduler, again, we've talked about the scheduler. It has this worldview, right? It wants to do an optimal placement. So it needs to walk through the entire database, and if you kind of think of compute nodes and their capabilities as a graph, it's kind of a complex thing that it does. But that aside, I think the real problem that we see with the scheduler right now is just that this dataset is held in the MySQL database. And so we get lock contention. We sometimes get spurious performance just because everything else in open-stack use is MySQL. So it is a horizontally scalable service. In practice, it's really a scale-up thing for those of us running open-stack in production today. We can throw a strong CPU at it. We can make sure that our DB is really big. Maybe we can use some of the ReedSlave functionality or some of the Galera, some sort of Galera sharding in some middleware to make sure that you hit an idle DB. But yeah, we have some problems in implementation that we're working on fixing. Conductor, if you're familiar with the conductor service, this is kind of introducing Grizzly. Conductor is an orchestrator. It does do some transactional operations. But the heavy lifting of conductor is really proxying a bunch of DB calls from Nova compute nodes. So anything that your compute nodes ask for is going through conductor.  So this is a major thing that we see problems with. This is a sizable amount of traffic that goes on here. Even in a baseline cloud that's doing nothing, you have periodic tasks running. You have resources reporting in. And so this thing is always handling a lot of traffic. So we do see issues with this. It's more related to our rabbit issues than the service itself. The service is horizontally scalable. You can blast it out to lots of nodes, lots of threads. It's a good design there. So moving on to Neutron. Neutron is kind of the exception in the core services of its API service doesn't just handle RESTful operations. It also handles some intra-component communication via the message bus. But Neutron server is horizontally scalable. You can run a bunch of them that's been that way since Havana or so, maybe a little post Havana. And then of course we have common agents here. We're going to have the L3 agent that handles routing. If you're running DVR, of course, you're in a fully distributed mode. But there's also a federated mode where you can have L3 agents running on multiple parts of your network, multiple network nodes. And then tenant routers and all those things that get scheduled by the L3 agents, they'll end up on different nodes that kind of distributes the load. So the one kind of got you here is that the placement capability is not that great. It's advancing, but it's fairly rudimentary at this point. L2 agent, this is what's responsible for doing all the wiring, the virtual wiring, so to speak. And it's a scale-up model, I don't know. Maybe it would be a problem if we were networking thousands of containers on a single compute node, but right now it's not really a big deal. DHP agent, again, you can run in a fully distributed fashion. Cinder, so we have a volume service, which again has a scale-out model. The caveat is what is your storage back-end? Do you have a storage back-end? Is it SEF? Is it some sort of NAS? Is it LVM? These are all gonna have different behaviors. The Cinder service itself, though, I mean, you can run multiple services. It's kind of a multi-worker model. For example, I have a buddy who runs SEF at a fairly large scale, at a company next door to mine. And he was having some problems with SEF. He kept noticing that the Cinder volume worker was blocking, and he came to me and asked me, you know, what do I do? Let's debug this. Well, we went into it, yeah, it was kind of one of those eventlet problems where a C library was doing a blocking call. Eventlet didn't know about it, so the whole thread was stuck. But the solution was just run more Cinder volume services. So we would run like 12, and everything was fine. Scheduler, similar problem, not as big a problem domain. So pretend to not see problems with that in production. Backup, again, just make sure this can be a service that handles a little bit more throughput. So if you're using it in your deployment, just make sure you have lots of RAM and that it's spread out the way it should be. API, horizontally scalable. Okay, Swift. Swift is kind of a dynamo, well, I shouldn't say dynamo-inspired, but dynamo-like object storage system has a nice DHT model, has a nice scale-out model. So to handle front end traffic, you have a proxy service, and this proxy service can be distributed over a wide range of servers and load balanced. And then you have container account and object services, and container and account, of course, are responsible for the mapping object, the actual storage. And it's a pretty simple model, right? The catch here is when you're adding more nodes, more storage nodes, to your cluster, or your replacing nodes, or hard drives, or whenever you mess with your cluster, you're essentially removing buckets and adding buckets to the distributed hash table. So the placement algorithm tries not to do this, but you will experience rebalancing when that happens. Glance, this was our image service. So the API, again, is one of those services you can run across multiple nodes. It has a little caveat, there's lots of throughput, images can be gigs and gigs, depending on what your workload is, where you work. So what's important is on these boxes, you need to have large amounts of memory. You don't want those images, you'd like to grab it, and then you'd like to keep it in the Linux kernel cache. So if you have 256 gigs of memory, and you have your 12 most common images in the Linux kernel cache, you can serve those straight to the network card. So yeah, that's kind of the only thing I would say about that.  So now we get to my favorites, these are the special mentions. So when I deployed my first OpenStack Cloud, ran into some MySQL problems. Looking back on the experience, I believe, again, they're mostly implementation issues. We have some concurrency issues, we don't have the best queries going to those databases, but when we talk about MySQL as a system, Randy already mentioned that a relational database, it's difficult to scale. It has a point of diminishing returns. When you're sending writes to all members of a cluster, eventually that write burden is just so large that it doesn't make sense. So we do see sort of an end to that scalability model. But for the OpenStack deployments of today, it's just fine, it's just fine. You use Galera, you use MySQL with standard asynchronous slaves and just load, if you're using Galera, use it in the active setup, load balance those queries, you're gonna be fine. But yeah, I mean, OpenStack's mission statement is that we want to be massively scalable. And massively scalable to me, well, so AT&T says they're gonna go to 300,000 nodes. That to me is massively scalable. So this may be something that changes in the future. Excuse me. Okay, now RabbitMQ, this is my ultimate favorite. Randy's gonna be sick of hearing this, but so RabbitMQ was a quaint choice when we were first developing OpenStack and we were playing around with it and we wanted to make something cool. I really, it really is at the root of a huge percentage of problems in OpenStack deployments today in production. What we've observed is that when you do the fully HAQs, active-active, like a three-member cluster where you're replicating all messages across all members of the cluster, it's just not workable when you get above about 200 compute nodes on a busy cloud. And so let's say you don't do that, don't replicate messages and just kind of have an active passive setup. Then you can get up to about 500, 700 nodes, but again, you're gonna run into the same wall. And I'm not blaming Rabbit. You know, there are multiple reasons why this happens. My thing is we shouldn't have a broker in the middle of a massively distributed system. There's no need for a broker. Direct communication, federated communication is the way to go. And that's changing. Oslo has a fairly mature 0MQ driver that came out in Metaca. It's development mature. It hasn't been used in production, but it has promise. So, all right, so conclusions from this talk. Here are the tactical things, all right? Understand that you have limits on performance. Understand that you have limits on scalability. You need to match these carefully with your application's needs. And you need to figure out where is the give and take, right? What are your SLAs, what are the real SLAs, okay? Invaluable is to have a performance pipeline. Since we have CI in the gate, and typically when we do development on OpenStack, we have CI in our local OpenStack. We don't see like syntax errors or a lot of the easier things happening. What we see are performance regressions and feature regressions. So, that pipeline's really important. And then, if you wanna have an easy time deploying OpenStack right now at scale, your option is kind of to use small clouds at this point. That's the easy way to do it. Nova cells could do, that would be, you need a team for that.  My sequel rabbit. But yeah, so we see improvements coming. Nova cells V2, that's coming. Really exciting and the messaging improvements that I already mentioned. And just to wrap up, I guess, with some longer-term thoughts or more strategic direction on scalability in OpenStack. You know, like Mike said, the mission statement of OpenStack is to be a massively scalable cloud. And I think what we're seeing is that you can reliably run a couple hundred nodes in a production environment. A lot of times people ask me, well, what does, you know, Mirantis has a distribution, of course, what does our scale lab certify? And so our scale lab tests up to 200 compute nodes at a density of 60 VMs, so 12,000 VMs. But that's a very specific configuration and it's not exactly a realistic runtime environment. We, the Mirantis reference architecture makes a lot of assumptions. It has a lot of opinions about how to deploy things. It doesn't use SDN, for instance. So your mileage may vary depending on your configuration. And going back to the comment about the CI CD pipeline, the performance testing tools in the OpenStack community rally and shaker. They're very good, very good test suite, but they don't always run a representative set of performance workloads. So for example, if you're a service provider and your workloads tend to be VNFs, very different performance profile than what most people would test in the lab with tools like rally and shaker, VNF workloads are gonna be extremely network-intensive. So you may be talking about using things like DPDK or SRIOV, they will likely be less intensive on other parts of the system compared to a general-purpose IT workload. So I think it's useful to have a distribution with a scale lab that says under this type of environment, this is what you can scale out to. But it is so important to do your own testing with your kind of workloads. And again, another comment Mike made, understand what your real performance requirements are. Is it important to be able to sustain 100 tenants booting VMs at the same time? Is it okay if that is slow, or do you really need that to respond in X number of minutes or X number of seconds? So it's very important to invest in your own. If you're going to try to push beyond that comfort zone of 50 to a couple hundred compute nodes, it's very important to do some investment in this area on your own. Because going back to more of the strategic points here, we know that beyond that comfort zone, there are some known bottlenecks, they are being addressed. But there are some fundamental questions about how to get really to the next level of massively scalable open stack clouds. There are some of those out there, CERN, Bluehost, a few others, but they're all very heavily custom engineered. And what we're seeing now is the trend is not towards investing a huge amount in one heavily custom engineered cloud. It's people looking to roll out a large number of clouds in different geographies to serve their customer base. And if you're rolling out 50 clouds or 100 clouds, you can't custom engineer every one of them. So there's some real fundamental questions that we're gonna keep working with the community with, and I think everybody in the community needs to participate in these discussions on how to get to that next level of very predictable scalability. So just a few comments, I guess, on what may be coming beyond the next release cycle. Mike mentioned there's NovaCells V2 coming in Newton, some messaging improvements. But beyond that, I think it's really interesting with all the buzz about containers. How might containers change this picture? So on the data plane, container workloads obviously have a lot of advantages for the application developer or the tenant VNF developer. Because containers are easy to scale out and have a lot of other nice properties. But from an OpenStack performance perspective, again, containers, if you're using an alternate scheduler like Kubernetes or Mesa's, they could potentially take some of the pressure off some of the known OpenStack bottlenecks like NovaScheduler and RabbitMQ. If those components only know about VMs and there's a second scheduler that's handling the greater number of containers, that's an interesting model. And it'll be interesting to see how that plays out as more people start doing things like that in production. But it poses a whole brand new set of questions. For instance, if you've invested a lot of effort in monitoring your data plane, collecting logs, collecting performance data from Solometer or using tools like StackLite to get some metrics out of it, that whole picture changes when containers are in the mix. Some projects like Magnum have some interfaces with Solometer, but they collect a different set of metrics. And if you're orchestrating, say, Kubernetes through Morano, you don't even get that much. So it poses a lot of different operational questions to think about. In terms of the underlay and the control plane, there's a lot of interest in containerized control planes and that will certainly help scalability in the sense that if you want to deploy five keystone controllers instead of three, it's really easy to do that with containers. Just spin up a couple of new containers. But fundamentally, there's parts of the control plane that aren't going to fit that microservices scaling model, MySQL, RabbitMQ, things like that, Neutron L3 agents, unless you're using an SDN that's more of a microservices compatible model. And again, there may be more of a direction towards federation. Nova cells to tear the scheduler and the message bus and the database, some other models like that. So that's another interesting area to look at. And this is again, I think we need as a community to really put some effort into this area because people who adopt OpenStack are going to not want to necessarily have to deploy a lot of small clouds. In some cases that makes sense, but then you need something on top of it. You need something on top of it that can orchestrate things that should be in common if you have images that have to be shared, if you want to collect metrics across all your clouds. So that's a choice that some people make. There was a talk I think yesterday about how to deploy a large number of small clouds, but then you have to invest in that management layer on top of it. So if you'd rather go the one big cloud model or a handful of really big clouds, as a community I think we need to start answering that question more. There's a lot of thoughts and a lot of ideas here, but there's nothing that's beyond the concrete things like Nova cells and some of the messaging improvements. There's more questions than concrete answers right now. So with that, I think we've got time for a few questions. Mike will answer all the hard ones. If you have questions, please step up to the mic. Any tuning that you recommend on the database I've heard, like on the Nova site, the records that could be cleaned up? There are. So queries tend to be very joint heavy. You tend to, so like your sort buffer, you want to have the right packet size to be able to track and transfer all that data. Those are the main ones that I would pay attention to. Also, in ODB write concurrency, IO threads, these are things that tend to not be tuned by default and they're pretty important. Yeah, and I think Mike mentioned this in his talk too, but with MySQL sometimes, if you're using regular monitoring tools, you may find that your database is still a bottleneck, even if the database nodes aren't running hot. And in that case, it could be a lock and tension problem. So one thing I think I've learned is have a good DBA who knows how to troubleshoot MySQL. That would be really helpful. I should mention something else. I gave a talk in Portland a few years ago and another way to just get rid of this is to distribute your reads and writes. The only native way to do that in OpenSec today is in Nova. It's actually code that I wrote, so I'm gonna promote it. So yeah, I mean, you can use a read slave and you can offload some of those reads to a slave cluster. And that'll limit your concurrency a bit. Yeah, and if you're a little bit more adventurous, I think with some custom health monitors in HA Proxy, you can distribute reloads across all the members of an active passive Galera cluster and still have writes just go to a single master, so. For questions? We covered everything. Very comprehensive. Okay, you've been a great audience. Thanks very much. Thank you very much.