Welcome to the show. It actually is a really really good Monday. It is. It's it's phenomenal So I I have a lot of callers on the on the call log if you want to call in the numbers eight four five two seven seven Nine one three one. So if you have a question call in to the show and I'll be happy to Answer your question Any questions posted on the YouTube feed? I won't be able to get to them for two reasons number one I would have to go in so close to be able to read it Nobody needs that much of a close-up of this. Okay, just saying and second Because the people on the call log have been waiting like 20 minutes to get through to me and I work better On an audit auditory basis like where I'm hearing things rather than Reading something and trying to type the answer back or answering the answer. So eight four five two seven seven Nine one three one. Thank you so much for joining me now. This show is running live YouTube so every Monday at noon I run live via YouTube for 30 minutes and Then it is sent over to WSCS Which is the TV show that I have so that it can be run live on WSCS I do have a second show create a life you love and since All of this has happened The studio is shut down which is why I'm doing the show this way I've gotten very casual By the way, it's casual Friday every day when I'm doing my show now It's okay. I'm happy with that. This is who I am. This is how like Workout clothes exercise pants. That's that's my jam. I just love that. Okay. So, um, I Yeah, I can bring my leg up on the chair. I'm very casual during my show now.  It has a bit of a different format I I may work with someone on the phone for create a life You love to do interviews which is how I did create a life you love in the past. I Had people who had Taken their path purpose turned it into their passion and made it their career or their side hustle and Interviewed them. Well, now I am Sometimes I'm going to be doing that but I'm going to talk about a lot about The dynamics of how do you get there? Because we all have a purpose. We all have a passion. We all have this thing that says My soul in my soul, this is what I really want in my soul I'm so passionate about this or I'm so passionate about that. So I'm going to talk a lot about What does it take to get Where you want to be what mind frames thoughts Programs and I'm going to work to help clear old mind frames Program so on and so forth one of them that I thought of this morning that I'm going to do a clearing for everybody right now and If you're new welcome, thank you for joining me During this show, I I do a lot of clearings or healings. I Believe it's really good to get to get an answer. Oh, you're going to meet somebody in June Okay, awesome But you know what it's going to be the same train record was before because we didn't work on ourselves We didn't heal what we needed to heal We are the same person but now we have more baggage from the last relationship or we go into the next job thinking It's going to be my boss is going to be the same. My co-workers are going to be the same I'm going to get used or abuse. So We take all of that with us. So in my shows, I like to do Some some healing so if you go back and listen to past shows There are so many healings or clearings about Finances love relationship. Those are all the biggest things What I'm going to clear right now is we believe things work out for the best for us and I'm definitely getting a no Oh, so we're going to clear that as anybody listening relistening. That's what we're going to do Okay, I want to get to the first caller Um because that is what this is about people calling in Asking one question if you are on speakerphone, please Go off of speakerphone Please your background noise will show up in the show and Nobody wants to hear that.  Please Get off speakerphone ask one question or one connection with a loved one on the other side I'm going to go first to Dun dun dun two seven Zero hey two seven zero. How are you today? I'm fine. How are you? I'm doing well. Thank you so much for asking. What's your name? Where are you calling from? This is Joy and I'm calling from Kentucky. Hey Joy. It's been a minute. Oh But your energy is always so sweet to get into I love it Okay, joy. What's your question or connection? I just have if you're watching on youtube or Do you as soon as your energy comes through?  Uh, what's up today? Joint custody petition and I'm kind of anxious and I wanted to okay, so The first thing I hear is good. Okay. Um Now good can mean fair But it can also mean some things. Okay. Some things you want are going to work out in your favor you have like justifiable Evidence or proof And it's going to help you However, don't be disappointed if you don't get everything you want. Okay Um And here's one thing I just I want to say, you know, um Well, that that's a whole show but basically Um I don't think I if I start saying what they want me to say that that will take that will encompass the rest of the show um So remember Everybody comes into everybody's life With a purpose and for a purpose so This agreement that you made long before you came in yada yada for this, uh situation living situation um It's it's folding out now. All you can do is kind of Keep your moral compass high and work within those parameters Um, that is going to be your saving grace through this that is going to be Uh How you need to handle this um Um The court date is going to go good There will be one little thing that's a bit disappointing but beyond that I think some things are going to go in your way that you're really passionate about. Um So just uh Acceptance is the key word here. Okay love Excellent I hope this was helpful and I can't wait to hear how it turns out I hope so now I have two questions for you. I see I you know I think in the past when you've been on the show your grandma has stepped in and I see somebody who's like a grandpa Who who is standing next to her that's who I am and he was really thin, right? What did you have a grandpa that was very thin? Okay, and he well especially next next to her she he was thin Now what I I see him in overalls like those Oh my god, so I'm so happy at first Listen at first sometimes what I see things. I'm like is my are you guys messing with me?  But I'm so happy I said that and then The thing I want to say is he's saying um You know for everything he went through in his life.  It goes on tv. I can't I can't say certain things We have to uh Say instead of saying it the way he's saying it Get your man on um Being through this and they're not only talking about tomorrow But they're talking about future dealings with this person behave as though you are a man and men they are sometimes Uh unapologetic. This is the way it is.  Have a good day I said bye Boom. I thank you. I said bye and then just turn your little booty around and walk away Leave them talking in the wind. Um, you don't have to be Nice or bend over backwards in this case. It's more important that you Focus on And the word they keep using is morality teaching your children The morality or morals to Bring everything to center. Okay. That's and that might not be as important right now as it's going to be later on But they're letting you know this is going to be very very important Good. Good. Good. Well joy. Thank you so much for calling. I'm so happy that I was able to talk with you Every time you hit the nail on the head. Oh, thank you so much. It's wonderful to get that confirmation um Yeah, the guides, you know people's angels and spirits are Are really really powerful And they will give you that peace of mind or that information that you really need to help make things Easier or better in your life So i'm happy that they are able to come through for you today and help you out and Give you that peace of mind and and that heaven stepped in to say hi and to Let you know they're watching out for you. They're they're both going to be with you um And you're going to you're going to have A little victory, but there is going to be something that you don't love completely So take the good with the bad is basically what they're saying. Okay, love Okay Here's the last thing is Do you know somebody named greg gregory on the other side? This person has been like Saying their name to me for like three days. Um greg or gregory Um, I don't know if this is associated with you or somebody else that's listening or going to listen Um, I don't okay. Well, then it must be for somebody else. So, uh, thank you again for calling in I'm gonna go to you. Oh, I'm so happy when I hear your voice. I just have to say Thank you so much for calling in again Keep me posted on how everything goes tomorrow and until next time make some miracles Okay, I'm going to go directly to the next caller that is eight five six Hello eight five six. How are you today? Hi, this is laura Hi, laura, where are you calling from? Awesome. How can I help you today? I just wanted to know I'm I'm wondering about A job or making a moving up. I'd like to know Okay, give me a second here You know laura, um You're the one I was getting this message for I think you're the one I was getting this message for earlier. Um I I don't know for sure, but um I'm I'm asking them questions right now So the first question I ask is are you going to make a move? Yes Are you going to move jobs or get go to a new job? And they're Like that's a Like you could if you wanted to if um If the right kind of like if the right thing comes along But one of the things I'm getting or I was getting earlier today and I've been getting it on a couple of my shows is um It's it's a really tricky time in our world And I have a lot of messages that come in about what's going on In our world, so I know The story behind the story. There's the story. They're pushing at us and then there's the story And anytime okay, I'm not I'm not going down that road today, but here's what I'm going to say um It's really important for you to know and You do everybody has this purpose everybody has this passion and Bringing some of that to the surface is so important for you right now so even if There's something And I this is the most read I'm just going to say it okay My mom god bless her I had created this cake a cannoli cake Now there are other cannoli cakes out there, but my mom's was very unique the way she made it the frosting Everything and she would she would decorate she used to bake cakes for an italian bakery. She used to bake Wedding cakes And she did all of this while she had her 40 plus hour a week job and children at home so the purpose of this story that they're saying I think is You have something inside of you that you need to bring out And do and maybe you have a specialty pie that you can make and sell for Easter or give away for Easter I don't know. I don't know what it is, but they keep hitting this cakes and pies cakes and pies Maybe it's because I let too much sugar go out of my body. I don't know I don't think it's personal, but I'm going to tell you They are really even at the beginning of the show when I was talking about my other show Purpose and passion are everything to our soul freedom purpose passion being able to Do that thing even if we don't know what it is at this moment. It's usually Something Come so naturally to you. You're like No, that can't be it I mean, I can't tell you the number of psychics That told me this is what I was supposed to do Before I started doing it professionally And I would be like, oh no that I it can't be that I've been doing this all my life It's just something I do and they're like, oh no, trust me.  Um As for a job again, they're going Maybe maybe not maybe maybe not Um, they want you to focus on what's inside of you Okay, what's inside of you And working Kind of on that for a minute and I I hope this makes sense for you. I know that wasn't The answer you were looking for but they're saying it's really the answer you need right now if you jump Into a job or another job You're going to end up where you were because you keep you keep doing the same thing you've always done And it's not fulfilling your soul So they really want you to look at what will fulfill What will fulfill your soul does that make sense to you? Well, I I had an idea of what would would do that What by the way side Well, um, unfortunately Perhaps the timing was wrong And so perhaps look at a new way to do it Perhaps if something was brought to you it's to be brought through you That's What I know That's been my experience Okay, love so revisit that and keep an open mind I'm tweaking it or tweaking the way you were doing it. They're saying Revisit that okay Well, it's for me. It's the story behind the story that you were talking about Okay What I'm talking about. Okay That's all right. Thank you. You're so welcome what I can say listen Just because it doesn't work the first time the first way doesn't mean it's not meant to be um Sometimes if you would have done it that way and this is with relationship with jobs with um schooling Sometimes if you would have gone down that road um The outcome wouldn't be as optimal As it's going to be In the new way the way you do it next. Okay, so again if you are on YouTube commenting. Thank you so much. Um, please call eight four five two seven seven nine one three one If you have a question, I'm going to get to as many people as I can today To answer their questions The next color is four one five Hey, Kate. How can I help you? Hey um, I definitely those some of those messages From you gave to her where for me and that always applies excellent Um But I want to say to everyone listening I uh testimonial I really recommend A private reading with you. Oh, it was an amazing experience. Thank you So much out of it. Thank you so much and I didn't pay her to say that I didn't even know this was fair when I clicked on her Names don't come up. Just I just Okay, thank you so much. Thank you. So Kate, what's your question? Let's get to you today. What do you want to know? um well believing that things will work out is For me it's about an appointment because it's a mess a complete disaster in california and My account is kind of on hold. Oh And um, so I'm starting kind of a pattern. I didn't get my last Check from them. It's a dependent So it kind of Panic, okay so again, um I'm just going to say this again, and I'm I know this is not a popular belief amongst some people um I know the story behind the story There's an agenda and it makes me livid and sad When I see what's going on to so many great, um countries. It's not only our country um Oh, and I just my mouth just wants to blurb it all out and I know I can't um So what I'm going to say is again, um Um I feel like it is going to go through just move quickly move very quickly Be accurate leave no No box unfilled. They're saying no stone stone unturned I feel like everything is going to come through very very quickly. I also though feel like this Okay I also feel like the sooner you can get out of there the better which can be difficult at this time However, sometimes you just have to take every Everything you can and jump when you can the longer you wait the more difficult it becomes I do I do truly truly feel I truly feel when you get to your new location What I say is something So it might be water under the bridge anyway. Well, I say go, you know what here's no, no, no, no, that's not that I'm saying do the steps you need to do Do those steps you need to do and then still plan and jump but what I you know, there is a place energetically That works for each one of us like in certain parts of the world or certain parts of The country that we live in we're energetically it's so much better for us and That energy of that area just invigorates us and the and trust me That can change you can be in let's say california and prosper for like Eight years and then after that eight year cycle all of a sudden it starts to dry up And it it's like chasing a relationship that was good in the beginning and waiting for it to come back It just it's not coming back. It's not coming back. It's not coming back that goodness isn't coming back So we have to sometimes jump to another area that actually Does have people working that actually does let us prosper that actually is open up like texas I'm not saying go to texas, but texas is opened up a number of states have completely Opened a hundred percent Congratulations, the rest of the governors need to get on that poop. Anyway Again before you blast me. I know the story behind the story What we're seeing We may be being tricked, okay Okay now Go and this is for you and anybody else Listening if it's not working where you are If you don't have your freedom if you are not able to make a viable income support your family pay your bills Go someplace you can That's that's not psychic. That's like common sense Do that Do that survival first Okay, having said that Fill out all the paperwork. Make sure you don't miss anything on the paperwork. It's really important to double triple check everything Everything is going to work out, but jump as soon as you can to that new Place jump as soon as you can to that new location And i'm going to say for everybody listening Belief is a very powerful thing belief Is extremely powerful What you believe is what you're going to bring to you We if anybody has and most of us have Know about the law of attraction or the law of the universe and our energy So on and so forth. So us being in a state of fear for as long as we have is changing Our whole world Get out of the fear if you weren't afraid. What would you do if you weren't afraid? Who would you be?


Your coca-cola bottler presents Claudia Claudia based on the play and novels by Rose Franken Brought to you transcribed Monday through Friday by your friendly neighbor who bottles coca-cola Relax and while you're listening refresh yourself Have a coke and now Claudia Certainly my guest Mrs. Meeker. Claudia. This is our cue to leave your beloved husband in the lurch Can't do that to me you're my partner I'm not our partnership has nothing to do with it Mrs. Meeker is the chairman of the presentation committee and she is nobody's partner Distinctly did not say Mrs. Norton. She said Mr. Norton. She did not say Norton and Killian She just said Norton that doesn't make any difference. We designed the Redbury Town Hall together You're breaking my heart come on Claudia. We can't stay at this opening ceremony all day We were both awarded the key to the city of Redbury. It isn't even a city. It's a township township city That doesn't matter, but you're you're a traitor and that does you can talk to Mrs. Meeker, too Isn't he awfully abusive this morning for an architect particularly an architect who was He's just been awarded a scroll and you were awarded a scroll too, and now you're unwilling to pay for it I haven't been asked to pay for it by anyone who counts got to take bluff back to New York Roger He can't wait for Mrs. Meeker to talk to David bluff can wait. You can take the train David We'll meet you if you two walk off and leave me here with this Oh, there you are, Mr. Norton.  I'm afraid to leave my registration book Certainly Mrs. Meeker how I'd be glad to if you leave me now Mr. Norton it must be so exciting to be an architect. I don't you have the most thrilling experience Well, I've never exactly think of them as thrilling Mrs. Meeker, but sometimes they are very satisfying Satisfying huh, what a beautiful way of putting it mr. Norton We here in Redbury are so thrilled with our new town hall that we can't help thinking how very very thrilling it must be for you He just lapped this up your husband doesn't they have to take out all his hat and have them stretch I don't see why we should wait any longer He isn't going to let that woman stop praising him during that poor dog sitting out in the car Starving to death, but he'd feel so much better if he knew that his master had just been given the keys to Redbury And Mr. Norton the excitement of seeing it all take shape under your hands It must make you feel just like a Michelangelo well Not all of the time mrs. Meeker the minute Mayor Collins announced that we were going to have this official opening I said we've got to give mr. Norton the keys to the city and a special scroll with flowers It was very generous of you mrs. Meeker There's nothing to the way we felt and you know mr. Norton you're really very young to be such a marvelous Roger I really don't see how David can expect us to stay here when he's having such a wonderful time Nobody's even talking about look at that smile on his face. I haven't seen him so happy since the day we lost two important commission David's only vice-rages. He just hates people to praise him and fuss over him and baby He'll grow up. Mr. Norton. I can't see why you should be such a hurry to go back to New York. Well mrs. Meeker well Well, I mean we we have a number of other things we have to do of building skyscrapers and things like that Oh, I wish you could stay this afternoon.  Meeker But you see I I do have to get back to New York and when you say lunch you you see I have of course it Isn't the guest house yet It's only a chicken coop, but I know you'd have the most original ideas for changing it over I'm afraid that's a little out of my line mrs. Meeker. You see my partner and I specialize in industrial architecture industrial architecture oh How romantic at least you will come to lunch and and tell me all about it, won't you by the way mrs.  Killian? He's here. You know and I think you'll find he has a lot more to say than I have No No, I haven't met your partner But surely mr. Norton you don't need each other just to eat lunch Oh, I think this would be a very good time mrs. Meeker for you to meet mr. Killian You'd find him a man of wide experience. You're much younger than he aren't you?  Where is he? Where did they go? What did you say? I said he's gone. He's disappeared. They can't do this to me Well after all mr. Norton he has a perfect right to go. Oh This works out perfectly Now you can come to my house for lunch. No, I'm afraid I can't I have to find mr. Killian But I've been expecting you After all you don't need him just to have lunch. Oh, I'm sorry. I do I must find him this minute and and my wife, too Yeah, why? Oh, they have the keys to my car. Oh, I have a car mr. Norton I suppose I ought to be very grateful to you two for waiting at all We weren't going to it was often mean of you to sneak out ahead of time and capture And it was also very mean of you to keep poor bluff waiting so long before his lunch Well, we take him we ought to celebrate a little don't you think champagne for us and a soup bone for blood I think it's a beautiful town hall and it's going to look even nicer in 200 years We really did do a pretty good job. They seemed to like it didn't they? Oh All mrs. Meeker was thinking about just now with those marvelous limestone columns. They aren't limestone anyway They're marble that funny. I thought I heard her say limestone when she gave you the scroll I think she did it But isn't it nice to know that even these slight inaccuracies don't prevent the audience from liking our work They didn't like it.  I Would have thought that you'd be satisfied just scratching their eyes out Can we take him anywhere name the place anywhere? We can go bluff can go to We've got a scroll that says we're honorary citizens of Redbury. They see that in that beautiful place They're on the right. That's the stone for him has the best onion soup in nutmeg County Let's drive in look bluff has never had onion soup. Well here is where he tried it Maybe we better leave him in the car until we find out if they'll take dogs even with the keys to Redbury Honestly, Roger. Nobody could say no to bluff such a nice dog But if you insist he stays in the car, I'm staying with him and if you stay in the car I'm staying with you.  You can get us in anywhere. That's so sweet. I'm driven to thanking you That's growl give you the right to kiss me before lunch. No, no, I really haven't read it all through Maybe it wasn't so silly to wait in the car that kiss was nice.  Not everyone is just that stone fought in They're known for their onion soup and stuffiness We'll find a much better place further off this time I'm going to go inside and tackle the innkeeper and I'll use the keys instead of the scroll By the way, I just remembered that in another few miles. We'll be coming to the coach wheeling onion soup Not exactly like a plan doesn't sound quite festive enough for the occasion But the people are a good deal more hospitable. We won't have any trouble with luck And he's not sure whether to believe it or not. Well, you just leave it to me Oh So that was the coach wheel in their manners were more like a steamroller I thought surely they'd allow us to bring in a dog Especially today. So you two men. I'd have as important as we three thought you were oh, we're important All right, it's just that nobody agrees with us. I'm ashamed it's happened in front of bluff He's going to be terribly disillusioned Claudia great day in strife undisillusioned. They're big enough so they can Stand a lot of it. The trouble is your dog is too big at least too big for an end We ought to have a minute or day that you could carry in your sleeves. We don't feed him He'll become a minute here. What are you going to do? He's hungry and so am I I know what we're not going to do We're not going to bring him to another celebration at Redbury Maybe not even you that big help or next time. We'll bring along a picnic lunch for the dog and you big help Isn't there another in a longer someplace? Yes. Yes, I know the place It's only about half a mile more to the River million Mega plan caveat. Oh I'm afraid our scrolls won't be much help there either. I'm afraid you're right All I can say is if somebody given bluff a scroll he'd do a lot better for us than we have for him Yeah, there's the River million Hey, David, aren't you gonna stop not even did you see that doorman? What's the next place? I see one where they're up ahead on the left that that that diner that greasy spoon They wouldn't even be able to read our scroll They wouldn't even have a lock to fit our Redbury keys in that case. Maybe they'll let us in Two jokes. This is a big celebration darling. We can't hold it in a little diner. Can we take bluff anywhere else? What makes you think you can get him in here? I'm going in not going in this time find out neither am I then why don't you send bluff in? See if he can get them to take us might work at that darling. Hmm. I can taste my Salisbury steak already Oh, you mean hamburger with with everything. This is a very special occasion. Well Here goes wish bluff luck Now bluff you've got to be very polite. Oh, yes show them. You know how to curl your little finger under Don't forget to tell them that I'm starving well It's now or never go on open the door Roger.  I wish we thought of this at the stone fork the food there is really much better You're not so fast Roger. We still haven't got in here yet. We're going to Listen This broadcast of Claudia was supervised and directed by William Brown Maloney When you hear your men folks talk about having a Coca-Cola cooler right on the job Does it ever occur to you that you have a white cooler in your own kitchen? Keep the family refrigerator well stocked with Coke and you can work refresh too Goodness knows you keep going hard enough all day long. So you'll find the pause that refreshes more than welcome Every day Monday through Friday Claudia comes to you transcribed with the best wishes of your friendly neighbor who bottles Coca-Cola So listen again tomorrow at the same time and now this is joking saying or if war And remember whoever you are Whatever you do wherever you may be When you think of refreshment think of Coca-Cola Or ice cold Coca-Cola makes any pause the pause that refreshes you


Now, the Iowa Trade and Association in Bios' state is calling on the governor, Dore Diri, to provide a land for the members in order for them to farm, which will in turn reduce the cost of rice, beans, yam, and other staples imported into the state. The group came up with this after an meeting with stakeholders on the association in Yanagua. Speaking to newsmen, Secretary of the Association of July, Omar, informed that currently are hijacking their goods on the highway, causing fear and panic among the members, which is also affecting their business. He urged the state government to provide modalities that would help make things easy for them. With the current tragedy that we are seeing, where people are looting and hijacking our goods along roads are adding fear among our members. Therefore, we come to a proposal that we hope it will assist and bring this hardship and inflation down. It is in our proposal that, based on our discussions with our members, there is a high level of willingness to key in a farming venture in the state. They show their readiness and willingness if a land can be provided for them to farm, because we notice what is happening in some state in the North as some of the Northern Governors say no more exploitation of food from their state and we have to have a poor security in this state. Therefore, I think it is in our resolution that we put a request for approval to the state government if a portion of land can be provided for us to farm. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


So, yeah, like I was saying, show hands who doesn't know who is eBay. What is eBay? Okay. We got that covered. So basically, we are under the umbrella of eBay, eBay classified group, basically focus on local markets, so websites where you can buy and sell all kinds of goods mostly or within your city or in your own country. So we have these 10 brands in multiple countries, like in Markplatz in Netherlands, Mobilde in Germany for selling cars, Ebecklen and Zeigen here in Germany. And we have our data centers mostly in Netherlands and Germany. So our cloud is basically two regions in those data centers, one in Amsterdam and another in Dusseldorf. These are some of the stats. So as you can see, I'm lying a little bit. It's not totally 80 cars, but almost there. So we have around 40 cars in each region in Amsterdam and Dusseldorf, 10K instances on Amsterdam, it's our biggest region, and around 6K instances in Dusseldorf. We provide volumes, object storage, block storage, yeah, I'll give it to Idrin. So as you know, at the start of this year, we were confronted with a really serious security bug, the Spectre ML don, vulnerabilities, and as you can imagine, our security stands needed to be covered. So you saw that we have a lot of repeatable brands, Mobilde, eBay, and then Markplatz. So we decided to immediately or as soon as possible patch against these vulnerabilities. Just a quick reminder for you that you don't know what Spectre Melldown is. So Melldown leaks kernel data to user mode programs, and Spectre is priming the branch predictor of the CPU and can predict what branch is going to be executed next. While Spectre is harder to exploit than Melldown, it's also harder to mitigate. I think there was even this morning article on Ars Technica where they found seven more bugs. So the gift that keeps on giving. So this is the short description of what the vulnerabilities are, and in the next slide my colleague is going to talk about investigating it. I will talk about our timeline and the phases that we have done for this project. We started, so the project basically started from January this year and ended in July. So there are, basically we started with an assessment phase where we're going to investigate what packages we need to patch, what we need to do for this to be covered. Then obviously, as soon as we started the project, we started the development phase where it developed all the scripts and automation that we needed to apply these patches. For Dusseldorf, our first cloud region that we patched, we took one month, and Amsterdam, then we took more than that, like around, yeah, so four months, yes. But in between we took more time in Amsterdam because in between we are confronted with some patches that we need to do in our own infrastructure, and I will be talking about more in depth what we did in those different phases. My colleague will talk now about the assessment phase. So we were not one of the companies that knew upfront about the exploits, so we got informed with the rest of the world. So we started to understand what mitigating the bug would mean. So for us, the path that we have chose to take was to update the Linux kernel, the QMU KVM, and then updating the BIOS. There were other paths. We saw that the colleagues from CERN took another path, but we decided that we want to offer maximum stability to our cloud and don't really need to rush into immediately patching with the microcode. Our central distribution was already a bit behind, so meanwhile, while we took the decision to go the BIOS way and wait for our Dell vendor to provide the stable BIOS, we also had time to develop updating Santos from an older version to an older version skipping one and then updating to Santos 7.4, actually. The first mitigations that we actually did were to provide as fast as possible, well, as fast as the distributions released to the public mitigations, and then we built our custom images, we worked with our tenants to test the images, see if they have any performance impact. They didn't, and then we launched internally the glance images for consumption. We were pleasantly surprised that Santos and Red Hat were among the first to have patches for this, so we immediately offered that, and then Ubuntu and Debian soon to follow with different repetitions and iterations of the images. While doing this, we also developed our strategy how to patch the cloud, and then my colleague is going to tell you what we developed and why. So it was clear that we needed to automate this big task of just restarting 1,000 iProvisers. So what we decided is to use Ansible as our main tool, and we used heavily Ansible roles which is a way to organize your tasks. As an example, we have open stack roles, hardware roles, update roles, and one of the most important ones is the checker role where you check the melter specter vulnerability. Things like resetting the eye drag or starting the compute, updating the operating system and the bias, those are roles that we could reutilize and organize. I will deep dive on the meltdown specter checker role. It's not that rocket science, so basically we check for the patched bias that it is the correct version that we need on the iProvisor. We check if there's the correct version of the kernel installed. We check if we have the KMU version that we need since it provides some CPU bits that you need to expose the system calls that are new for patching the vulnerability. And then in the end, we run a checker on the host that is an open source script that basically tests the variants that we want to check. You can even find it on GitHub. It's a very nice script that covers everything. This is a very simple example of a Nessival playbook running where everything went okay and in the end, so you can see the versions that we were testing, the SVs that were mitigated, so it's pretty, so while you're running, it's pretty easy to detect if everything's all right. Now going deeper on the playbook that actually patches the iProvisor, so I divided this in three main phases, the pre-tasks where you just prepare the iProvisor for patching where you disable puppets, in our case, unmount the file system, ZFS, maybe check the current bios and stop, obviously, the instances that you are running on the iProvisor, then the actual update happens where you upgrade the bios, you update the operating system, we are actually upgrading the bios on the operating system, so that's why it's done before. And then we go on to the post-tasks where we reboot compute nodes and we run the checker that I just showed, and if everything is all right, we just start and putting back the iProvisor in the previous state. So mounting the file system, starting the services, starting canaries that I will talk later where they are, starting the VMs and enable the compute node. So as you saw here on this patching playbook, there's something that maybe needs clarification, so some services were restarted. A very important service that we need to restart is the VRouter agent, so in eBay Classified Group we are using Contrail for our STN, so we need to restart our VRouter agents, basically the Contrail component that forwards packets from the VMs. We also have a canary running on every iProvisor that does a little bit of monitoring and testing of the actual iProvisor, and we also were unmounting and mounting the ZFS file system that we use to host the actual VitoMachines. Now we will talk a little bit more of other tasks. So we are fortunate enough to mainly have cattle in our cloud and not necessarily pets, so one of the things that we were fortunate enough to do was to actually disable the iProvisor and completely stop all of the VMs, and then our tenants had the choice either to spawn their workload in a different zone or have the downtime with us, and then we are going to just bring up their VMs. So we didn't use any live migration, that was one of the things because we thought that live migration is going to possibly impact the maintenance, so we wanted to keep it as simple as possible and then ensure that the VMs are up again. So we had to develop a way to keep the list of the VMs running on the host, so we didn't do nobody evacuate, and then actually something very important for us because we're not living in a perfect world. We had broken iProvisers where they were not suitable for production, but they were in some sort of pre-maintenance task where we wanted to change some disks or something, and then we wanted to save the disabled reasons for them, so whenever you do Nova Service List you would see why that iProvisor was not productive, so we wanted to save that reason and restore it for future use. The BIOS upgrade, yeah, it was not perfect again. What we actually did, we downloaded the Linux provided package from Dell, and then using Ansible we executed it locally, parsed the output of the script of the package, and then made the decision if the operating system is safe to reboot, and after that we have rebooted the iProvisor. There were cases where iDRAC got stuck. Our maintenance didn't have the task to update iDRACs and make sure that iDRACs are fully functional. We just thought that as a precaution we would just reboot the iDRAC and then try the patching again, and in some cases again lessons learned, we had to do the patching manually. Bruno is going to talk about hardware failures. So with all of these reboots that we did in our infrastructure, very often the machines would not come up with or a BIOS was corrupted, and we needed to go directly to the machine and actually install the BIOS, no network or CPU memory errors, so because when you are booting exactly an intensive task that will cause some failures if the hardware is not in a good state. So there's always a risk when you reboot that you probably will decrease your capacity. I will move now into talking about our testing phase. A milestone was when I sat in our team chat in Slack, we have meltdown fixed computes, which means we had like a playbook that actually was running all the steps. We had all the versions and it was checking that it's fully patched. We selected some group of users to do this testing so we didn't patch everything at once. We just added some infrastructure with those patches and see how it behaved. And at the same time we were keeping like an eye on the community to see if there was some consequences of patching the iProvisers, if the load would increase immensely, because this was one of the people were afraid that this could happen. I will talk a little bit about as well, so while we are doing this, we are starting all our infrastructure and we use AVI network load balancer and we needed to do some automation on it. So the way that AV load balancer works is that there's some service engines that are basically VMs that are spawned in each tenant and those VMs basically do the load balancing. They are small load balancers. So we needed to migrate all of those service engines on the zone or rack that we were upgrading, because at the point that the version that we had was not aware of the availability zones. So they were not like spread around on different availability zones, even if we just did upgrade on that one. We automated this with AV as well as STK and some Python scripts. So there is a lot of tools to do this as well, which works very well. Adrian, we will talk now about the campaign in Dusseldorf. Yeah. So Dusseldorf, like we said in the beginning, was the first data center that we decided to patch. We went the full cloud way where we told our users that we are going to take down one availability zone per week and they were fine with that. We never had such scale maintenance, so it was a learning curve for us as well. So what we did is for in a specific day of, I think it was a Thursday, we started, we disabled complete zone four. Again our users could have taken the downtime with us or moved their workloads in the other three availability zones. We took the zone completely down and then up again, patched everything and restored the VMs. We did the same for zone number three and then for the last two zones we got a bit of momentum and then we did them in the same week. The lessons that we learned were that, yes, we can do it very fast. We can upgrade the entire data center in a few weeks, but there was not that optimal. Our users, so we found we had feedback from our users that the elastic clusters, elastic search clusters were re-syncing shards a lot, so if we weren't careful enough on bringing up entire racks that they had problems with rebalancing the elastic clusters. So for the next campaign in Amsterdam, we took that feedback and then we had another strategy. Yeah, so this data center took more time. We started with the same approach, so one zone per week. On the first upgrade, we had an issue, which is that we needed to patch our STN. I will talk more about it in details on how. That happened as well while we did the second zone, so we had two patches in between. That's why it stretched a little bit on our target to do it at least in one month. So we started with one zone per day and then we finished with one rock per day. I will tell you why we did this. So there was two patches that we needed to, that we detect while we were booting. One was related with our control STN and the other with the AVI as a service. So control has a protocol that distributes configuration from the configuration nodes to the control nodes. We needed to apply a patch that basically was catching an exception, and yeah, it was delivered by Juniper control and back port. The same happened with the AVI load balancer as a service, where a service engine was having trouble in setting up a cluster interface, and we also got fixed from RV to fix the old and new creation of service engines. So yeah, it took some trouble to do this. Now Adrian will talk a little bit about the performance. So Dusseldorf was our test case, our first test case. Our German colleagues would know that this is the elastic search cluster of a repeatable website here in Germany, eBay, Clientsagon, and then the lower graph is the hypervisor load before, during, and after we executed the maintenance. And then above is a bit more detailed CPU instructions and usage, again before, during, and after the maintenance. For this specific use case, which is one of the most important things, so imagine everyone searches for the things that they want to buy on eBay, Clientsagon. We see actually no impact for us. So everything was more or less in the same limits before and after the patching for spectrum meltdown. So for this example only, we can say that we weren't impacted by the 10, 15% that the community was saying. This is the same case for Amsterdam. So this platform is a dual data center platform. And then this is the same observation. Before, during, and after our maintenance, we can't clearly have a pattern where performance got impacted. Yeah. Just to go a little bit about, maybe this can be a little bit confusing. There's a lot of noise, at least on the first one. Only the conclusion is here is like, if there is no change before and after, so they all stay the same. Even if you can see maybe a little bit increase of load, this is just people creating more instances and the app provider gets more loaded. So there was no difference between the patches. I will talk a little bit about maintenance strategies. So we started with one zone per week, but then we ended up with a rack per day because it looked like a good compromise between velocity and impact. Like my colleague said, some platforms, which is a group of users for us, were kind of afraid that it was impacting too much on their workload. Like taking down a whole zone will be too much for them. So we were doing like one rack per day. We were also notifying which VMs were affected on that rack. So people need to know what's going on. This is something that we need to automate better. And we were as well communicating all the steps to our users. So we used Slack and we were just giving them updates on what was happening. And we were stopping the VMs when we were updating them and when everything is finished. Because usually users would want to know, like, what's happening at what time and when they will get the VMs back. So what we have learned, Ansible is a great tool for infrastructure automation and we encourage everyone to use it with roles or any other modules that you can use. Do not rush an updating as soon as a vulnerability is discovered. Try to hold it off until you are sure that the patches are not causing any consequences to your infrastructure. Because like that happened where a BIOS update was released and then it needed to be pulled out because it was rebooting the servers. And the same with the microcode, it was pulled out. And again, so make sure that what you are patching is really on prime. And so I talked about the patches that we did. It's actually good that we restarted our old infrastructure because you can catch either hardware failures or catch bugs on it that you only do when you are stressing the actual infrastructure. And try to scope the maintenance as the best to reduce impact, like we did from reducing from one whole zone to a small rack. That's it. Questions? Yeah, I have a question. Regarding that you update your BIOS that you said you did it via this in the Linux command line, then you have to reboot server. How do you deal with these VMs actually on the hypervisor? Are they cloud ready or did you have to live migrate those VMs before the rebooting? So like my colleague said, we just stop them and leave them there. We upgrade and then we start them. So they are not migrated. So those applications actually they are not affected by the rebooting? Yeah, we basically instruct our users to replicate their application towards all ability zones so they can be aware of failures. So their apps should be aware of those failures and cope with them. Thank you. Can I just... Sorry. I've got the mic I'm louder. Just a question. You mentioned in your last slide about lessons learned, like restart your whole infrastructure often. Is that something that you currently do or is that like a lessons learned? So you are implementing something like chaos monkeys as a result of this to sort of try and tackle that. Yeah, good question. So we don't do that currently. So we are doing it right now when it's needed like a new feature or new upgrade, but we intend to do it every three months. So we are thinking next year to come up with a plan to automate all of this. Yeah. Thanks very much. Cheers. The ladder. Is there any specific reason for not using massive live migration? Because you said you did not live migrate except AVI machines. Yeah. So we did a test with our users and then we saw that some of their workloads were negatively affected by live migration. So we decided together with them that they should take just the downtime because we have like maybe I didn't mention it correctly, we have four zones in each data center and then they can still have enough capacity in the other three zones that are up. And what we notice also is depending on the workload, live migrations, maybe 80% of them are passing and then maybe 20% are not correctly live migrated. And then we didn't want to stretch out the maintenance. We didn't want it to be very long. So we wanted to do a zone in one day ideally. So this is why we didn't want to do live migration. We have pretty beefy nodes and then live migrating everything was going to take some time, even on 10G. But did you use live migration in a regular basis or? Depends again. So we tend to encourage our users not to treat their VMs as pets. So we most often tell them either destroy your VM or we will take it down or something like that. You're lucky enough to have cloud compliance clients. Yes. Yes. That's true. Yeah. Hey, I saw all the steps, there were seven steps including disabling compute node, you know, stop services, start services and batch and then do the BIOS update. So all these are done in a single and simple playbook or you have several playbooks that you're achieving. So yeah, it started like that with a big one, which was easier for our testing. But then we noticed that we needed to split them up. So we did some splitting like things that we need to do before, things that we need to do only for updating. So we split it and it will actually better because we tried as well to have different team members. So it would be easier to split the playbooks and they will run specific tasks. So this leads where there is no continuous end to end patching for a node, right? So you have to wait until one playbook finishes and then start the next one. That's like a manual intervention. Yes. So it's true, but we can also control it better what's happening. So we had the two possibilities. We had this whole playbook that runs everything, but then we realized that since we had more, we were doing only one zone, that was a lot of hypervisors to patch. So that's why we split it. But then when we moved to a rack-based maintenance, we could actually run the whole playbook from top to bottom. Okay. Yeah, we've been using the same solution and some of the playbooks for our patching too, but then now we have stitched all our playbooks using RCIC redeployments so that it's just a trigger and then it'll notify you if there's something wrong happen during the run. Yeah, that will be the next step in the optimal. We are as well, we want to do that. Yeah. Just a suggestion. Yeah. One more question. So for your BIOS update, you mentioned you have experienced corruption. Did you figure out what is calling that corruption? Oh, yeah. So basically, for some reason, some of the idracks were performing poorly. So nothing would help. We did a rack reset. So just basically rebooting the idracks. We did everything to have, let's say, a stable idrack, but none of our solutions in the end gave us 100% stability of the idrack operating system. So the next step would have been to actually update the idrack firmer and then update BIOS. And that would introduce a new dependency in our past cycle. So you're saying basically just updating the BIOS independent, you leave the other components alone? Correct. Correct. We have solutions where we can upgrade every firmer of the compute node, but that was again out of the scope. We wanted to keep it very, very targeted to fixing very fast this vulnerability. So are you downloading those BIOS updates from locally to the server or where you update that? Right. So in the playbook, we basically do a VGAT. We place the file on the hypervisor. We execute the Linux binary and then parse the output. And based on the output, we decided that compute node is stable to reboot. And in most of the cases, I think we had just a few cases where corruption occurred. We were right about ... So when you downloaded those BIOS images, you verified those images are good? Correct. Correct. Yeah. So the image was 100% correct, the Linux binary. And then we applied it. Yeah. Sure. Okay. Thanks. We verified that. How diverse is the hardware you have? Is it 100% Dell or do you have HPEs and Lenovo's? And did you have to write in a custom code for Ansible like modules to deal with some of the BMCs? So we have homogeneous hardware from one vendor from Dell. And no, we didn't have to write anything for the BMC. We know later on while we were doing this project, we saw that there is Ansible for Redfish and that could do what we wanted actually. But again, our iDrax were not all at the same level. So this is why we didn't want the Redfish part. As a later project, after we did the patching, we upgraded all of our iDrax. So now we can use Redfish. What we wrote was a way in Ansible Playbook to reset the iDrax. So if it fails, we would run again. So that sometimes fixes because there's some jobs that are stuck and we resetting them, we could apply. But in the end, there was some corner cases that we needed to do it manually. I think if there's no more questions, that's it for us. Thank you. Thank you.


We have to know the truth if we want these children being napalm. He's the publisher that wants to show us the cost of... We are all mausoleum effect, mausoleum people to be silenced.


Okay, so today we're going to talk about capacity management and provisioning. The kind of subtext is the cloud's full, you can't build here. There's kind of a riff on the scene from Forest Gump where he's trying to find a seat on the bus and no one's going to allow him to sit down. This is kind of the experience that we've had at Rackstreet with the public cloud in terms of what has happened capacity-wise that prevents instances from building. My name's Andy Hill, Joel Prizes with me and Matt Van Wickle. So the public cloud capacity at Rackspace, we've deployed over 100 cells in the last two years. We're constantly, we usually have five or six cells in flight. We're constantly adding new capacity to every region. Our process used to take systems engineer three to five weeks after getting the gear handed over to them with a bare metal install. And now we've taken the process from three to five weeks for a skilled engineer down to the operator that's on shift is adding capacity and the new cells are coming up and it's little as one day but usually it takes around a week for a new cell to come online and the restraint there is around networking. Top of Rack network configuration kind of has some issues around automation. We still don't have the right pieces in place to make that as well oiled of a machine as the rest of our provisioning that we do. Around the control plane sizing, so like our NovaDB, our NovaScheduler, the sizing of those nodes that we run, we kind of have to have some ideas around the data plane and its impact on those nodes. So for example, if we have a very, very wide data plane that's doing lots and lots of downloads and uploads of images from Glance, we have to scale out our Glance architecture. And then the other kind of impact with the number of compute nodes or the size of our data plane, the breadth of the data plane, I guess, is how large should our NovaDB be? If we have 600 hosts for a given cell and each one of those could have up to 25 instances on it and the instances are turning all the time, that's a ton of records in the NovaDB, right? So we have to kind of pay close attention to the sizing of our cells in terms of the number of compute nodes that are involved and then the sizing of the control plane components that orchestrate that data plane. The other kind of sizing considerations that we make for cells are around private addressing space. So if you think about one of those very large cells that I just mentioned, it could be up to a slash 17 for the private addressing space and that's a really large broadcast domain for the instance traffic. And once you get to broadcast domains that are around that size, it gets kind of interesting on the open V-switch side of things, to say the least. We have a presentation later on today that's going to get into detail on that, but yeah, shameless plug, right? We're talking about that more of the details around it, but that's just one of the kind of considerations that we have for our build-outs of the cell sizing. And then there's kind of the unsung, underappreciated consideration for your cell control plane sizing and that's more around overhead and complexity. So we don't have infinite resources at our hands for the control plane. We have to right-size everything. And you can't just add nodes and services and different components to each cell because that's more complexity for the operators, that's more complexity for the employers, that's just more complexity overall. So we really try to right-size with the size of the cell and its nodes, minimize that so that the control planes that we spin up are as cheap as they can be. And as simple as they can be as well. Some of the hypervisor provisioning kind of considerations that we have to make are around cow images, so like copy and write images. If you're familiar, if you're kind of coming from a VMware world, it's the snapshot of a VM that kind of hung around forever and ended up filling your drive. You have to kind of plan for some overhead with ZenServer. That cow image size can be two times the size of the disk, so that can really be painful. And then sometimes there are problems with cleaning up these snapshots afterward. So if we're using software to create the snapshots and to clean up the snapshots, sometimes those fail and can lead to scenarios where the disks are full on the hypervisors themselves. We also do pre-caching of images at RecSpace. So the standard images that you get whenever you sign up for a RecSpace, like your Ubuntu image or Fedora image, those kinds of images we call them base images, and we cache those directly on the hypervisors themselves to make the build times as fast as possible. Well, that's also some drive space overhead that we have to account for whenever we're sizing everything out. Otherwise, we could fill up the drive just like we were talking about. So with the combination of the pre-cache images and the cow overhead, we can end up having some interesting scenarios that you should really be aware of for sizing out your hypervisors. And these are just the Nova config options for that pre-caching option. And, okay, so the other kind of things to be cognizant of whenever you're thinking about sizing out a cell and its control plan and those kinds of things. Sometimes pretty awful things happen with the hypervisors that are being used, right? So you need to have some excess capacity for emergencies. If you don't have a spare chassis to put customers on whenever an existing chassis crashed, that's a really bad situation. So there's some options within NovaCells to set a reserve. The other thing is that you're kind of bound for a cell. So it's not like we can just move you to another cell that has capacity. Cells itself doesn't support moving instances from one cell to another. And then kind of the final, like, gotcha that we encountered from a, how do we size our data plane and how do we size our control plane was around VM overhead. So there's a tiny amount of overhead that each VM has that's directly related to stuff like the number of VCPUs that it has, the number of disks that are attached, all that kind of stuff. And however tightly you're trying to pack your hypervisors, you're trying to maximize those hypervisors, this overhead can really start to matter and be an important factor. So there's now accounting for that overhead in the scheduler and there's links to the patches that landed. This is all available today. And I'm going to hand over to Joel for the bigger problems that we've seen in production. Hey, everybody, I'm Joel. I'm going to be talking about, as Andy mentioned, some of the kind of big gotchas that we get when we're trying to scale out our infrastructure to meet the needs that we see. There are way more than we could probably cover in this, in the time that we have here, but these are some of the main ones that we run into. So specifically around load balancers, and that could kind of also be expanded out to load balancers, firewalls, just any kind of single point you might have, say between cells in your regional control plane or the regional control plane and just your ingress. Oh, you went ahead real fast. Glance and Swift is a big thing from the control plane standpoint, just in terms of pure networking, that's where you're going to be generating the vast majority of your bandwidth, and so that's where you hit a scaling point on that side of the house before pretty much anything else in terms of running the cloud. Fraud and non-payment instances. If you are not running a public cloud, if you either are deploying private clouds or it's just something for internal use, this might not come up. But when you start getting into a large public deployment, it becomes a major deal for capacity. Routes, routes, routes. I'll get into that a little bit, just routes, so much routes. And row testing, and by that I mean testing this new capacity that you're adding and how to adequately do that. So load balancers. Yeah, it can become a very quick single point of failure or at least a single point that you can easily saturate things. So up there we have alternate routes needed for high bandwidth operations. Specifically that pretty much refers to glance and glance traffic for us. We have an alternate routing path that we can go through that circumvents these devices for us, specifically for glance uploads and downloads, so that we are not saturating those. It slows things down when we do, obviously, and then the other shared services that are going through those connections can start to fail, like your database connections and that kind of stuff. That's another one that we have listed up there specifically, database queries. If you were in the talk before that was in this room where they did a pretty good deep dive of some very nasty queries that are coming in. We've seen some pretty bad scaling issues where unoptimized database queries were sending way more traffic through load balancers than they should have been to the point that we were saturating or near saturating those connections. A lot of really good work was done to sort of catch a lot of those and bring that data down, but that's something you have to be really aware of when you're adding new capacity and especially if you're deploying to multiple environments. We've seen cases where things look great and we're doing it in some of our smaller regions and then we moved on to the larger ones and it was like, this was an exponential growth for this particular problem and that step up, we crossed the tipping point and moving up. You really have to keep an eye out in your pre-production and when you're analyzing potential changes coming down the pike that you're looking at increase per instance for things like database traffic so that you can accurately plan around that. Swifting glass bandwidth, I talk about this quite a bit. Again, single bottleneck. One thing we do to keep an eye on this so we know when we need to add capacity to handle this is we have triggers based around build times or alerting based on build times and imaging times. If we see those start to slow down then we know that we're hitting some type of scaling point in load for swift uploads and downloads.  We don't share the cash between them. So say like if you have 20 Glance API nodes that are handling your image downloads, if 10 of them have an image and 10 don't you have a 50% cash hit ratio. It would be a pretty neat feature to be able to fill the Glance API nodes to themselves talk and share that knowledge among them so that if one or two of them have an image cache that's say pretty popular, like it could be a customer's image, a customer one, not one of your base images, something like that, it could defer to one of the nodes that does have it so that you're not using bandwidth you might not need to by reaching out to Swift or whatever your image store is. We need to get image downloads out of path, or sorry, that should be, well, we need to get image transfers out of path if we can and by that I mean using the Glance API node is essentially a proxy for your traffic to Swift from the hypervisors can be a real pain. You're making a single point that you're concatenating basically all of your traffic through for those transfers. You can run in some really interesting problems there and I'll get into that kind of towards the end of the slide here. We try to avoid these things by caching the base images as Andy talked about the, you know, the base images are, you sign up for Rackspace, you do an image list, those images, if we're rolling out a new Glance API node we'd be sure to at least seed it with all of those images so it's ready to go as soon as it goes into production. We also try to precede images to hypervisors ahead of time if possible and we saw some really big improvements on that in some specific use cases. For instance, our big data is a service offering, Hadoop offering uses a very limited set of images they have and so we made 100% sure that all of their hypervisors have their images on them when they're good to go and that was a dramatic improvement for their build times, obviously. Just everything was good to go and we're using the fast cloning for a Zen server which you can read about on the wiki there so that makes it, the image provisioning part becomes almost near instant for those at that point. It's basically just make a new cow and you're good to go. For troubleshooting problems on this side, Glance and Swift and I guess just overall OpenStack projects aren't as great as they could be at sharing request IDs when you're trying to track those problems down can become pretty interesting. It would be really great if you had an actual single request ID that you could use throughout the entire stack but having to switch between this request ID and then going over to the Swift transaction ID which can be different to try and find where your bottlenecks are here can become very difficult and that makes thus planning for adequate capacity moving forward much more difficult. So what happens if you can't scale out anymore? That means you have to horizontally scaling specifically. We reached a point when we were relying almost entirely on horizontal scaling for our Glance API nodes and as Andy mentioned, unfortunately we don't have an unlimited capacity or unlimited resources to throw at things and we reached a point where we couldn't do that anymore. We couldn't just add more Glance API nodes like we were doing because we were running into saturation problems at the next layers up on the networking side. So we had to actually re-architect and move over to new hardware entirely using 10-gig mix. We had to start spreading across to different segments of our network to spread that traffic to different aggers, different top of racks. You have to really keep an eye on that. Otherwise, you're going to start impacting other services not just Swift and Glance specifically. So fraud and non-payments. Fraud, how do we handle those at Rackspace? We immediately marked them as suspended because the account is flagged as fraud. And that's great. It stops somebody from being able to use those resources that you're providing for free. However, it still takes up capacity while they're around. So what do you do about that?  About a year and a half ago, two years ago, something like that, we developed an internal tool called Account Action Year which basically looks at our internal auth system, our user base, and looks at the accounts that are flagged as fraud. And it has a set of business logic in there to say, hey, this is fraud. It was marked as fraud X amount of time ago. We are confident at this point that this was in fact real fraud and we're not just suspending and deleting somebody's instance. That shouldn't be. I'm going to go ahead and clean this up. So our rules for that on fraud are pretty aggressive. Like if we have a whole team devoted to deciding if somebody is legitimate, whether it's person or not, and we take their word for it more or less. So once something is marked as fraud, it's pretty aggressively deleted so that it's not taking up space in our cloud. Nonpayment is very similar to fraud, but it's worse in terms of capacity management. Because if it's nonpayment, that means it got past the fraud stage probably and at some point they did give us money and we're a company and we like money quite a bit actually. It's almost one of our favorite things. And so we want them to give us money in a point where somebody might have gone through a rough spot and they couldn't make their bill but they want to. We want them to come back into the fold so to speak. So we try to give them much more time to come back into being into rack space basically. So it's a very similar account action here. We're still handling those. It just has a different set of rules that's much more lenient in terms of turnaround time before it deletes the instance. But again from capacity management for quite a bit longer. This is where something like shelving instances might be a nice feature to have or that we could leverage so that we could completely shelve the instances someplace and they're not taking up that capacity. But then say several months down the road if somebody decides they want to come back like hey, yeah, we still got that instance. Sure, welcome back. Road testing. You have to aggressively test any new hypervisors you throw out there. We just bootstrapped something and assume it worked and hoped for the best. So how do we do that? With the new cell it's pretty easy. We use bypass URLs and by that we mean cell specific API nodes. So those are basically configured like how many people in here are actually using cells? All right. Us up on the stage and our friends in the front row. Yeah. We call the bypass URL cell specific API nodes are probably more similar to what the rest of you who don't use cells see. It's just an API node that doesn't know it's in cells and it's built to directly talk to the cell that we're provisioning. So when we're doing our testing for that before we link that cell up to our regional control plane we build using that API node so that bypasses the region. There's no way that customers can inadvertently build to that cell in any way. We're finished testing and we link it up to the region. We delete that cell specific API node because once we're on the region we can use scheduler hints to send to that cell directly. We're looking at potentially moving forward using cell tenant restrictions so we could link the cell up entirely but then have it set up so that only RQE tenants basically could build to it. And that would allow us to more fully test the entire pipeline for building to the cell and making sure everything is working. This is a lot harder when you're re-adding capacity to an existing cell. Sometimes we don't provision an entire cell at once. We get certain cabs in certain time and we put them on as soon as we can. It might just be a new cab that got delivered or it could be a re-kick like if a host we failed out for hardware problems rectified and we're putting it that hardware back in how do we test to that? That cell is already linked up to the region. It might be in production already. We can't stop or want to stop customers from building to that cell as a whole and there's no way to disable a compute node or a hypervisor but still be able to test build to it. We can disable it obviously but then our test builds fail. That makes things a lot trickier because we want to test to that node and we want to test with it in the cell that it's in and everything but we can't prevent customers from potentially being able to build to it. We've got a huge amount of builds coming in even if you just flip it on just for one second and try and get your build in and then close it back down. Odds are pretty good that somebody might land on there. That's a real interesting challenge for us. Matt Stuff. So these guys just to put in perspective do most of the heavy lifting. I'm the BDI manager type but even capacity, planning and management with respect to OpenStack and business has a few tricks to it at that level specifically around the supply chain and how you sort of handle this ever flowing stream of boxes of different configurations. Some of our own decisions from a product development standpoint and how that changes the way we plan capacity and ultimately there's still some challenges from upstream that the guys touched on and we'll visit you here to get in a second. So when you start dealing with public cloud or even internally if you start to say expand to more and more groups if you're running an internal cloud you start to get different constraints around capacity planning. My favorite is the large customer requests hey I'm a sales guy and I know that this customer is going to need but you have to sort of look at those kinds of requests and you have to think about them in terms of overall capacity over subscriptions those sorts of things and sometimes that does actually impact the speed at which you pull in your supply chain expect delivery of cabinets and how you sort of plan what's being built out next and in what regions or what areas of the deployment. We have a lot of triggers we look at the easiest one is the percent used in a given space you know or 70% okay it's probably time to order. The trickier one at least when you're dealing with finance departments and some of the other folks that you sometimes have to deal with is the largest number of slots available so again we offer a range of sizes in any of our flavor classes some of them include whole host sort of you can have an instance that takes up all the RAM on the host effectively. In those particular flavor classes it's very important to track the amount of those empty hosts that are available regardless of percentage because we can actually run into problems where in say a new region or a smaller region we may only be at 56% capacity but there's no room for the largest instances to build in that flavor class and that's just as bad as being out of capacity altogether. So that's the trickier one but we try to keep an eye on it and then ultimately we're also a public cloud provider so you know IPv4 addresses become an interesting constraint to deploy capacity we'd love a lot more but that's kind of hard to do these days so we're still you know as Aaron sort of wraps up its allocation for example in the US the process of getting more is very stringent so we're sort of stuck waiting on the company to negotiate its next round of IP addresses both for us and for our dedicated business and sometimes that means we're like watching some graphs get dangerously close to the numbers we don't like while we have gear sitting there and all we need is the public IP addresses to drop on it to sort of support the customers so that'll get more interesting as time goes on obviously the other big thing about it is schedulers the sales and scheduler services in NOVA aren't aware of IP addresses as a constraint for scheduling builds that's something we're trying to start to talk to people about because that's actually probably more useful for us right now than the RAM typically we run into more problems from IP addresses and provisioning than we do from RAM and provisioning then we have a couple of services we've built, auditor and resolver some of that work we've talked with there's a small project around in the community called entropy we've talked with those guys a little bit that does some of the same functionality but basically these are services that help us keep an eye on various aspects of the environment and in some cases will take action the simplest one I can describe is we actually have an ability to weight down a cell for builds as soon as IP thresholds drop below a certain amount it'll actually go in and update that information now we're still fighting for an actual disable flag because if you've ever well the old use cells but cell weighting is kind of magic voodoo that doesn't always, it's not like an on or off it's a relative value and so you can try to attempt to not build to one and yet it still wins because it has the most available RAM and these guys alluded to this but this is kind of an interesting aspect to how we manage capacity our control plane actually runs on an open stack installation itself so we actually stand up a small private cloud if you will and in there we build all the instances we build instances that become the control plane for the public cloud so when we talk about horizontal scaling we literally were just spinning up more and more instances until we sort of saturated the top rack switches that supported that little environment and to do the glance expansion we actually went out and stole performant hardware from the customer fleet and made them into bare metal boxes and we're now circling back and making other changes for that alright so from a product management perspective we're always trying new things I think in the last just over a year about 13 months or so we've added four completely new flavor classes we've renamed two of those but each one of those brought with them a completely different hardware footprint which meant a different VM density which meant a different amount of sizing per cell and a different number of cabinets per agar and so you start to sort of I think we have six or seven different types of flavors that we manage across all the different regions and so that's just more variables you have to play with also we're constantly doing code deploys I mean our goal is to try to stay relatively close to trunk usually within a few weeks I want to say the one we're testing right now is from mid-October so we're not too far off but again that has kind of its own pace of deployment and then you have all these new capacity coming along and so there's always that period of time you have to sync and make sure that okay we started this cell we deployed a new version of the region that we go back and patch this cell before we flip it on and cause all kinds of havoc even within even within our different hardware classes we have multiple vendors and so sometimes we actually find subtle differences in using hardware from vendor X versus vendor Y and in our oldest sort of it's called standard our oldest flavor class we actually have three separate vendors and in two of the cases we have multiple revisions of the hardware from that vendor now typically that doesn't hurt us at the nova level but from a total capacity management standpoint we do find quirks along the way so it ends up meaning all of our tooling has to sort of be able to determine like which of the seven hardware variants is this and then of course non-production environments additionally take strain on capacity from a not only hardware resources but time and effort and tooling spent getting those up in competition with the production environments so upstream like I said earlier a disabled flag for cells a lot of our upstream needs really circle around cells there's actually three sessions going on this design summit specific to cells one today and two tomorrow the nova team is looking to sort of feature complete it possibly even make it the default in sort of a no op single cell don't have to worry about it kind of model but disabled disabled flag for the cells is big also for the host like the guys talked about just being able to manage like an admin only flag or a disabled flag that allows me to build tests to a specific host without sort of exposing it to customers before testing scheduling based on IP capacity that I talked about and just overall the son the feature completion pieces I know it's being pushed very heavily by Michael still and quite a few of our core devs are involved in those discussions I think we're going to be sitting in them today and tomorrow to kind of make sure from an operators perspective those things see their way through but I think in order to give folks time for questions is that the last one yeah we have a couple minutes left so any questions it's it's most likely disable host yes but we've had trouble with so availability zones as part of the feature completion for cells so this is wrapped up in the first class citizen stuff for cells we can disable the host and we can deploy to it and do everything we want to set it up but then it's difficult to then test that while it's still disabled without exposing it to customers to also build to even if you do a scheduler hint even directed at it it will fail if it's disabled so we can flip it on do a like a full host build to it real quick and hope that we get in there first but yes but if the flag is set to disabled it fails for us it comes back with a scheduler failure I'd like to know more but at the same time we want a full fledged VM because we also test a bunch of stuff inside the VM so we would prefer to have an admin disabled so we could do a full complete I mean that's that's what we would like to yeah our goal is around road testing the entire node making sure all the networking and everything for the instance comes up properly we can do some stuff to get around it and get a build there and do it safely but like we would we prefer to be supported and be trunk basically worth and you're not trying to do anything funky we would just like to say only let admin build to this basically would be ideal so like number of nodes in a cell versus the amount of control plane nodes we put to them yeah so cells are going to vary we have any some that run around 100 up to some that are like 600 hosts or hypervisors a lot of that depends on which type of hardware they are for that we run what is it we have cells, scheduler, couple databases we actually don't run all of the opensack services on a single node we run a separate node for Nova scheduler a separate node for the various components right it comes down to we run quarter of magnitude so I'd say for I think our target right now for most of our cells we're trying to get to the database pairs that are about 8 gig instances running in our private cloud and then the scheduler in cell services run as like 4s or 8s as well and I think that's pretty much all you need for the cell itself now where we I'd say we don't have the math completely right as the sort of I won't say auto scaling but the scaling the APIs and the glance nodes and stuff in conjunction we're still trying to we do it over time but I don't think we have an exact math that says when I get to 15 cells I need 12 API nodes yeah and it's also kind of a function of the use of the cell and and what happens over time so for instance one cell may have a bunch of instances that are booted and then they stay online forever and then another cell may have a bunch of instances that are booted and deleted which ends up really adding some database constraints to it because it keeps all those deleted rows so in conjunction with all of this we have to come up with a consistent pruning plan to kind of give us an assurance that the database nodes are under similar load and so that we can size them consistently all that stuff so that we're yeah I give you a specific example because we were looking after the last talk on database performance we've just recently pruned our global database in two of our regions and in both cases just keeping 90 days of deleted instance information means I have several hundred thousand deleted instances in those databases so it's it's a good problem to have right but it's it does pose interesting challenges when you start to get to those kinds of things so not yet host aggregates is one of those features that we get when they finally feature complete cells I actually like the idea of host aggregates for a couple possible reasons it might allow us to actually start mixing flavor classes and or hardware vendor variations within the same cell and be able to control some live migraine and some other really touchy things that want stuff to be pretty exact also the testing pieces may become a lot easier because you can start to say well these cabs are already in production these are new we're adding them to an existing cell we're gonna throw them in their own host aggregate and route builds based on that so we don't today but I'm actually pretty excited about the potential for using host aggregates especially in the capacity management space so they're internal right now we call them auditor and resolver we are exploring how we can action here action here that's another one we literally just looks at our account services feed coming out of our yeah the open source tool entropy which kind of mirrors some of the auditor resolver functionality and we're looking at a couple other projects where we might be bringing some of this code to these these are all just things that we've built really in the last six to nine months except for action here and so we're still at that phase where we're getting it right for ourselves and then understanding which projects they line up most closely with upstream anything else awesome well thank y'all for coming if you have any questions you can catch us up here


Ita ni,a kwaила kuha kwaimu kuha islandu ngadi kulia wekuja kwaipa kuminii kwaiku na kwaiba kwaiba kwaibu kwaizi kwaiduматu wa kama ambili kuный lozi kwaibu kamao kwaiba Jumbopolis kwaibu kamao kwaibu kwaiba kwaiba kwaibi kwaibu kwaibu Felo countrymen and women, this is what the colonial government did to the nationalist leaders. They did it to Dr. Kaunda Meisorist in peace, they did it to Mr. Simon Masaka Puepue Meisorist in peace and the other nationalist leaders that were fighting for a cause. I'm not comparing ourselves a new generation of political leaders to the nationalist leaders but that was a script for the colonial government. The question that I pose to President Aga Indehichilema and his new dawn administration and his inspector general of police, are you now the new colonialists in our country? That you seem to be following their script? Because the transferring, the restricting of the nationalist leaders to different parts of the country was to break their morale and their spirit, to give them mental torture and break them for them to give up on a cause. This does not make you look good. President Aga Indehichilema, as well as the government, any person across the country who has a complaint against Jackson Silavue, they must follow Jackson's days and lodge a complaint from there. It is retrogressive and it is embarrassing on the part of government. If something ever happened, for example, to Tyali, God forbid, or to Mr. Rafael Nakachinda, who President Aga Indehichilema bear responsibility if anything tragic happened, that's where cup zambians. This is not the zambia we want. This is not the zambia the zambian people voted for. We need much more than this. I thank you.


I appreciate that phone call, but I'm going to have to splatter my back to the back problem. I appreciate that very much. Well, please do, I'm sorry for the occasion. Exactly, exactly. Well, Mr. President, first of all, I'm very grateful for the opportunity to be able to see and talk with you in person. We've had some exchange hands, county officials, that the new federalism initiative not being banned. I know in the old college you needed to be back in late August, and you said that you had not been in it. We're going to continue along the same line, but it is essential to us, politically essential to us, both Republican and Democrat, that we have some dead minds. We enjoyed it too. No, no, I was thinking that. I don't see it. I think. We need a few more students in those years. Actually, this is from that moment to later. That's what happened today. And we ought to put them at the end of the day. Welcome, and I know you're best at things you want to talk about. So who's on first? We'll have Margaret to start on. Thank you, Mr. President. Mr. President, I would like to speak to a very brief overview, very honest analysis of the conditions that were made to be pieces of legislation. And that will always be. I think it's safe to say that we are totally together on one opinion, and that is that the cities are in terrible trouble. And I am personally led by the conclusion, although perhaps some would argue that we're doing better in facing new reductions. We represent some 14%, but we're only the exact same. The budget, and we have 33% of the cuts. And we plenty of us have supported the reductions, the budget cuts to a certain extent because we realize that what's good is good for the cities. Thank you.


Question is from Julie Cassandra. How do you alter your diet or training versus? How do you alter your diet on training versus non-training days? I love this. Yeah, I feel like we're gonna have a discussion around this, because I think you and I are a little different. Yeah, well, I really love to manipulate this. Like so, and I do this all the time with weekends especially, I've tracked my movement, my activity, my workouts so consistently for so long that I've really got a pretty good idea. So even though I'm not tracking right now, I have a good idea of like what a Saturday looks like or what a good Saturday looks like, what a bad Saturday looks like. And what I mean by good and bad is by my activity level. Like, was I very active that day? Did I train like I need to in order to get my movement and my calorie burn up on a Saturday or Sunday in comparison to like a Monday where I'm starting my day really early and I'm moving a lot and I normally get my workouts always. So I like to adjust my calories that way. I mean, we already talk about the benefits of undulating calories anyways and not having like this linear calorie intake where it's just like you always, your diet says you eat 13 or 1500 calories or whatever and so every day you do that. I just like to break it up. Like I think and it makes the most sense to eat less food on the days when I feel like I need the less food because I'm not moving and I'm not pushing the body. It's an off day of working out. So that's what I'll do on my training. On my training days, I make sure I'm fed. I make sure that I'm taking care of my body if I'm going to push up on calories a little bit. I just had this the other night with Katrina, we were talking, this was like two nights ago and I knew I needed to get at least another like two meals in and it was already like six o'clock because I was just behind on calories and I had a really good hard training session and yes, I could not eat and get benefits of burning more calories, but I'm trying to build at the moment and I want to make sure that I reap the benefits of a hard training session like that. So I want to be fed. Now, if I hadn't trained that day and I'm still behind calories and I would actually not worry about it. I would probably skip a meal and be low calorie that day because the need for the extra calories is less important when I haven't sent a signal to build muscle. If I just sent a loud signal to build muscle, that in my opinion is one of the best times to make sure that I'm getting adequate calories and protein versus a day where I don't train at all, a day like that I might skip a meal. Yeah, see, this is where I think you and I are different. By the way, I don't think there's a wrong answer here. So the way Adam's doing it, perfectly fine. If that's your preference, I think that's a great strategy and a lot of what you said makes perfect sense. Now for me, it's actually, believe it or not, it's a little bit of the reverse.  I like to have a lighter feel. I like to, I'm a little busier typically on the days that I work out. So I don't feel like I want to eat as much. On the days off, I like to eat a little bit more. I feel like it helps me recover on the days in between. And here's the other thing that I've, and this is my own theory, but when you're sedentary, you are, you're sending an atrophy signal to your muscle, even if you worked out the day before. So, and again, you can test this yourself. You can work out super hard today, and then stay in bed for two days and see that you actually end up losing muscle, even though you worked out really hard, you know, the day before. And I used to do this as a kid. I used to think sitting around would help me recover faster. Later on, I realized I actually got to move because I have to continue to send that muscle building signal. Now one way to offset that, and this is again, my theory is to keep the calories high and keep the protein high. High calories, high protein, by themselves, not always, okay, and this is not like a great strategy all the time, but in combination with exercise, I think it works great. When your calories are high in your protein side, by itself, independent of workout and anything else, it does send a anabolic signal. In fact, if you take the average person, don't change anything about their lifestyle at all, and just bump their calories in a protein, you'll see a little bit, a tiny bit of muscle mass gain and strength gain.  So when I'm sedentary eating more, it feels to me like I am able to send more of that signal. So I would do this also when I was competing and I was really consistent. And I too don't, I like this strategy. I don't coach this way though because I know the average person doesn't have the same self-discipline as I do. And what I've noticed with clients is that if I give them the okay to feed more when they're sedentary. Oh, they just have more access to food. Right, they tend to make worse choices. Great point. And so I agree with everything you're saying, and I too will manipulate and play around with my food like that too. And I think it's great to go both ways if you have the control and discipline to do both. But behaviorally speaking, I noticed that clients, if I tell them on Saturday that they can have a surplus of calories, they have a harder time controlling it versus telling them, hey, you're not moving nowhere near as much. So we don't need to feed as much. So this is a day we actually need to reduce. And so I'm telling them to reduce knowing too that they still might kind of overeat a little bit. So I tend to push that direction. No, you're 100% right because connecting my activity today to my food intake today is a great strategy to help people monitor their food intake.  More cravings at that point. That's why, yeah, for me personally, like I definitely tend to not eat as much if I'm sedentary just because it's just, I've conditioned myself to not seek it out as much because it was always craving type foods that I tended to want to gravitate towards with that mentality. But yeah, I do like going into workouts not with a lot of calories. I like to go into the workout but then replenishing after the workout, I tend to eat quite a bit. Yeah, so for those of you who have the discipline to do this, I love to feed myself a lot on the days before. I have hard workouts when I'm more sedentary. So I do that. Then when I go into my workout, I have less food but I did eat the day before, right? And I have great workouts. Typically good pumps and I'm really strong. So I used to always prepare for my workout with my food intake the day before rather than the day of because I like the excess calories, the carbs, but I don't like the feeling like I just ate. So I like to go in with the slightly fasted feeling but also have the energy from the day before.


All permanent MOOCs on the Virtual Language Six Campus consist of a fixed number of units where the first one, the PMOOC preliminaries, informs you about how to use the class and what it means to study online on the Virtual Language Six Campus. And the last one is the optional PMOOC evaluation, where you can give us your opinion about the class. Central to our PMOOCs is the idea that you have access to all units of a class at any time. However, if you wish you can also define your own fixed rhythm using the class timer. Let's say you want to go through the class on a weekly basis, then select the seven days rhythm. If this is too short for you, go for the ten days rhythm. If it's too long, select, say, a four days rhythm. Please note that the rhythm is just a guideline for you. It marks the unit you should work with in red and makes available the practical solutions in time. Whenever the red marker moves on, you will see these clickable symbols on the practical sheet. But note that, of course, you are free to change your rhythm whenever you like and you can always override it at any time and access any unit of your choice and the components within it.