Welcome to our show, presented to you by Code Pink, Woman for Peace. Okay, so here are some warm-up questions, you know the rules, press the bottom if you know the answer, name this world famous villain. That is Luisa Almagro, the Secretary General of the Organization for the American State. That is correct, ladies and gentlemen! And that is what today's contest is all about. We asked the public for ten reasons why Almagro has two in the row. Participant number two, go to the back and we'll start with participant number one. Michelle, give me five reasons why Almagro has two in the row. I love this song. Yeah, Almagro and the OAS lead the fuse for the 2019 Cuenote. Correct, ladies and gentlemen! The OAS without basis, question the results that favored Evo Morales' reelection, setting off unrest and activating a plot that over through him. Almagro immediately recognized the cool government which committed summery execution and widespread repression. Great, so now give us your second reason. Yeah, he helped legitimize four more years of the narco dictatorship led by Juan Orlando Hernández de Moldova. That is correct, the 2017 elections in Honduras were actually written by fraud. And initially, Almagro and the OAS did the right thing, they denounced the fraud and called for new elections. But after the Trump administration celebrated the results, Almagro backtracked. And despite the documented crimes of Juan Orlando Hernández's regime, he embraced and legitimized the Honduran government. Now number three, yeah, Almagro continued the OAS long history of interference in Haiti. It's in Haiti again correct in 2020 when President Jovenel Moise ruled without a parliament and gave himself an extra year in his term. The OAS issued a press release telling Haitians they should comply. Next. Yeah, he embraced the 2016 coup and the Temer regime in Brazil. Brazil 2016 coup. Wow, Michelle, again correct. Almagro had nothing to say about the coup against Dilma Rousseff and was quick to accept the Temer regime visiting his government less than two months after the coup. Also, when former president Lula was arrested in 2018 and barred from the upcoming elections, Almagro and the OAS did nothing. Paving the way for the right wing extremists, Jalbo Sonardo, to win the presidency. And last, but not least, yeah, Almagro ignored human rights abuses by security forces during massive protests in Colombia, Haiti, Ecuador, Honduras, Chile. It was human rights abuses again correct. Yes, silence when it comes to Colombia, but he even congratulated the government of Lenín Moreno in Ecuador for its handling of the protests that left 11 dead over a thousand injured. And hundreds arrested. He will later say that former Chilean president officially defended the public order. The same Piñera who declared war on his country's protesters and whose police forces targeted their eyes. Michelle, you did an excellent job. Now let's bring a picture to the camera. Okay, ready? So give me your five reasons. Go. He's waging a hybrid war against Venezuela. Venezuela, that is correct, Michelle. Under his watch, the OAS violated its own charters and procedures time and time again in an attempt to intervene in Venezuela. He even recognized fake interim president, Juan Guaidó, and accepted his fake diplomats into the OAS, although Venezuela had formally left the organization by then. Almagro also tried to invoke the Rio Treaty, a defense pact that could have opened a path for a regional invasion of Venezuela. Next, he's one of the main drivers behind the attempts at regime change in Nicaragua. Nicaragua, that's correct. The violence of the protesters who killed at least 60 people, including 22 police, as well as government officials and supporters, was deliberately ignored to frame a narrative and justify an intervention. Next. I have one. He lashes out of the media and punishes those who criticize him. Lashes out at the media exactly like Trump and Bolsonaro. Almagro published a bizarre rant that invoked Nazis, attacked scholars, and made absurd allegations against the New York Times. He denounced two OAS electoral observers as spies for disagreeing with his false claims of fraud in the 2019 Bolivia elections. He's corrupt. He's corrupt. He is likely corrupt. Hard to argue with that. In 2018, Juan Jiménez Mayor, a spokesperson for the mission to support the fight against corruption and impunity in Honduras, resigned from his post because of the lack of institutional support from Almagro and alleged corruption in the anti-corruption campaign. Next. Yeah, well, apparently he has nothing to say about the statue of Queen Isabella in front of the OAS. Statue of Queen Isabella in front of the OAS, that is correct. At times when governments all over Latin America are taking down statues that pay tribute to perpetrators of genocide and racism and colonialism, Almagro has ignored. A July 2021 request from Code Pink to meet to discuss removing this symbol of centuries-long holocaust unleashed on the indigenous people of the Americas. Ladies and gentlemen, what a match. Like nothing we've seen before with such bright intestines. Oh, wait, wait. Oh, okay. So I've been told that the public has more reasons why Almagro has to go. So go ahead. Put them in the comments. Post them on Twitter. Use the hashtag Almagro Renusia Almagro Resigned and tell us what are your reasons to be proved that Almagro has to...


Hi students, we are going to talk about international compensation in this topic and in the subsequent few topics that are coming after this. In this topic, we are going to discuss about introduction to international compensation. You know that whenever a concept of HRM goes on to the international level, the challenges of it becomes higher. You know that the process of that particular aspect becomes more complex. So similarly, international compensation also becomes a more complex process in the international scenario. So let's take a look at what is international compensation and what are the major challenges and issues regarding that. When compensation is about domestic compensation, it is something which is usually seen as a cash payment that is made to people who are employed in an organization and the bonus and benefits associated with that. But when it becomes an international issue, then there are a number of additional aspects which are attached with the process of compensation. It's not just a cash value that you pay to your employees, it is something that needs to be looked at from various different aspects, from aspects of equity, from aspects of culture, from aspects of national boundaries, from aspects of what kind of rules and taxes are applied in host countries and how does it compare with the parent country and how to equate all these different aspects of compensation. So when you are compensating people in an international context, that becomes a bigger challenge for the human resource department. And therefore, it also becomes a mechanism for achieving certain organizational objectives beyond the objectives of only compensating the people. So international compensation, it has increasingly been seen as, number one, a mechanism to develop and reinforce a global corporate culture.  It also becomes a mechanism and primary source of corporate control. You are able to control the behaviors and actions of your employees through the compensation and you are able to control their behavior internationally. And then it has become a mechanism of explicitly linking performance outcomes with associated costs. So it is a complex process to actually link performance with associated costs of internationalization. So it has become an increasingly higher mechanism to explicitly link performance outcomes with associated costs. And finally, one of the most important considerations that come under compensation is that it is increasingly seen as a nexus of increasingly strident, sophisticated and public discourses of central issues of corporate governance in the international context. So compensation because it involves a number of ethical and corporate governance issues and when it goes to the international context, then it is a matter of debate, a matter of discussion how people from various different cultures, from various different nationalities, they should be compensated and therefore a lot of debate has started to arise out of this discussion that how people are compensated and different nationalities and culture and whether it is fair and just and or there needs to be some adjustments. For example, there is a lot of criticism on the disproportionate payments, the disproportionate salaries of higher level, top level management, particularly in the American U.S. multinationals that the higher level, top level management, they are paid in a disproportionate amount of figures and it is criticized that what is something special that they do that they are paid in such great numbers and people who are at the lower levels of the organization, they are not paid in the same proportion. So it and because it is something which is then taken to the international context so people make comparisons and then they discuss it and they talk about the philosophy of how people should be compensated in this international context. So it has become a nexus of sophisticated debate regarding corporate governance in the international context. As we just discussed in the start of the topic that international compensation is a much more complex issue than the domestic compensation and how is it more complex? Because there are a number of things which are which need to be taken into account. There are a number of things which are added into the compensation packages. There are a number of things which the employee must be compensated for when they are sent into an international context. Sometimes it is because you want to induce them to take the assignment. It's sometimes because the cost of living of the international assignment is so high or sometimes it's because the environment of the host country is so challenging that you need to include a lot of different aspects of compensation in the international context. So the type of compensation package, the elements of that compensation package would include pay and then this pay would include taxes as well as cost of living. Then you also need to take care of the housing of employees that are going on international assignments. Then you need to also make sure about the safety. Usually in the domestic environment you don't have to take the measures for safety of your employees, safety in the environment in which they are working. But in the international context in certain hardship locations you also have to make sure that safety is ensured where they are working. Then transportation, similarly education of children usually when you are paying people in the domestic environment, you don't have to take care of the education of children. You don't have to look for their schools. You don't have to look for appropriate facilities or how much is it going to cost something that is given to the employees and they are supposed to manage it themselves. But in the international context, usually the international organizations, the multinationals, they take care of this aspect of the education of children. And then length of stay has to be then equated with what type of compensation package has to be provided to an international expatriate. The international compensation is becoming more and more complex because of a number of issues. Number one is that there is a growing need or growing use of outsourced activities because a number of human resource activities, they are outsourced from different organizations, from different places, from different companies. For example, many organizations, they outsource many American organizations, European organizations, they outsource their call centers in India or some of them they also in Pakistan. So compensation has to be then equated according to the culture in which the people are working as well as the corporate culture of the organization which is hiring them. So people in the call centers usually get a very good pay as compared to the developing country environment. For example, in India or Pakistan, which usually people get payments, in the call center, although it is not a complex work, it does not need a higher level of skill, but it is something because you are hiring multinationals, so you get a good pay in the call centers. So it is something which needs to be equated, which needs to be balanced and the companies who are hiring in the host countries also need to see it and the government and social structures in the host countries also need to see how to balance and equate it. So it is a complex phenomenon. Then another important complexity which involves in international compensation is balancing centralization and decentralization of incentives. So when you have a subsidiary working in a host country, so you need to make sure that whether the incentives and bonuses which are paid to the host country nationals and the parent country nationals which are working in that subsidiary, it needs to be centralized. The headquarters need to give incentives or the incentives need to be decided by subsidiary management. So how much centralization needs to be done on these matters. Then balancing need for more accurate performance metrics on international assignees. So it is something which is a very complex issue that how can you measure the performance of international assignees because the performance of international assignees of expatriates, it's something which includes a number of factors which we have discussed many times in previous topics particularly adjustment to the new culture and getting things done in a new culture is something which needs to be equated in the performance metrics, in the performance management system. So one of the complexity of international compensation is that you need to actually measure the performance in a way so that it accurately reflects how successful a person has been in carrying out that international assignment and therefore compensate them accordingly. Another aspect of complexity is the universal applicability of disproportionate executive compensation packages. So it is something which I have just discussed that there is a lot of debate on this fact that disproportionate pays are paid to the executive level and this is something which needs to be taken care of, it needs to be discussed, it needs to be debated upon and an equitable treatment and compensation of people has to be employed but it is something which is not done at the moment, the executive level people they are getting disproportionate payments and that is something which is a complex matter to deal with and then there is a greater need for transparency due to increasingly available information through internet and social media. So when you go to host countries, different nationalities, so usually when this internet and social media was not available, you would not get the information that how much money people are getting and what payments are they getting but now that the world is so well connected, so because of that, comparisons are made that Pakistan is getting so much information, Singapore is getting so much, Australia is getting so much and then they equate it and compare it, so you need to keep your systems more transparent so that you can give your employees an answer to how you are compensating your employees. So there is a greater need for transparency and that creates a challenge for designing the compensation packages and therefore increases the complexity of international compensation. So in this topic we have discussed about the various different aspects of complexity regarding international compensation in an international scenario.


Hey guys, welcome back to my YouTube channel. This is Daniel Rosal here today talking about one of my favorite subjects, increasingly my obsession. And this is videography. I'm one of the most important aspects of videography being collecting audio from the field. And I wanted to talk about a microphone I picked up earlier this year. And that is, I'm going to put up my screen here, the Shure PGA 48, which is a microphone from Shure that comes in at only $41 on Amazon, which seems like a steal, right? Shure is a quality manufacturer. And I don't think that Shure make bad microphones. I'm pretty sure that even there or this was my rationale for purchasing two of these guys was, you know, well, even a basic Shure microphone is going to be a solid thing to have in my audio bag. Now, the reason I bought two of these was because as I'm getting into videography and this YouTube channel and so on and so forth, I wanted to have a two microphones so that I could do a two person interview. Let's say two people sitting on a couch, each person has a microphone.  You don't actually even need a mixer in order to capture two wired XLR mics directly into the camcorder. So that's what that's what I bought two of these microphones for. Now, they're actually a really, really good. It's a really good microphone and I've done a couple of videos where I have sort of sound comparisons and even a couple of my friends who are involved in the world of videography and photography. The only time people have asked me what microphone are you using was actually when I did interviews with the Shure PGA48 and they were kind of shocked that a $40 microphone delivers such decent sound quality. Now, the title of this video is going to be Shure PGA48, good for an interview mic or something like that. And I'm going to explain why, unfortunately, for handheld and interview application, it's not the right microphone. But before I do that, I want to just show you what a better choice might be. It's something that I kind of didn't really pay a lot of attention to when I was ordering this microphone is I thought, well, I need an XLR microphone. I need something that looks like a reporter's mic or a handheld mic or an interview mic. And I think I probably put something like kind of, you know, XLR dynamic microphone into Amazon and just found this one from Shure. And I was like, that's great. But you actually have to pay a little bit of attention to how the microphones are marketed, right? So this says dynamic microphone, handheld mic for vocals. And that's kind of the key word that you need to be on the lookout for. Now, if you think about it, a microphone that's intended for vocals, right? Let's say a stage microphone, you've got someone singing into it, let's just say, right? So that microphone is going to be sitting the whole night in a some sort of a mic holder, right? Therefore, handling noise doesn't really need to be a big concern because it's not going to be actually handheld, right? It's going to be sitting in a stand. Now, if we go into the Shure PJ48 and this is what I would do if I were looking for this microphone again, is you can actually go into this very useful feature in Amazon where you can search product info, Q&A and reviews here. And I just popped in interview. Now, the first reviews actually positive. They said it's great for red carpet interviews. I'm a bit surprised about that. I feel like if you're shooting celebrity interviews or red carpet interviews, it's probably worth using a microphone that's a bit better than a $40 mic. But if you get to the third and fourth one, you can see here, since I plan to use a mic for handheld interviews, I found it necessary to to return it. However, if you're simply putting this on a mic stand and talking into it, then you'll be fine. Now, by the way, this would actually be fine for podcasting. You can see the way I'm using this microphone, the Audio Technica AT2020. It doesn't have handling noise. But if it did, it wouldn't be a concern because it's just stationary. Now, this is the telling review. This one here from Hayden Anderson. This person says, I bought this for street interviews and I cannot use it for such due to the handling noise. I'm not sure what I'm going to do with this microphone. Maybe I'll use it in the studio on a stand. Let me show you guys the handling noise to which these people are referring. This is the interview I shot just the other day with Noga, who is someone involved in my industry and we talked about impact investing. And you can see the setup I have is I shot this on the Canon XA40. I actually got my wife to work as the camera operator and she did a great job. And we're just using one microphone. So this is actually the Shure PJ48 in my hand. And I'm just passing it over to Noga every time she speaks.  Now, when I actually edited this interview on Friday, I had to literally cut out every single mic pass so that handling noise didn't come into the final cut, so I'm going to shut up and listen to the audio. In response to that sort of broad question, where are we with impact investing in Israel right now in well, almost 2023? OK, so first of all, thank you for having me. OK, so you can hear firstly, you know, the microphone audio quality I think is pretty decent, but it's just this place here. If I go over the timeline, 2023, OK, and you hear that noise as I pass the microphone out. You know, you could do what I did in post-production and just edit this out. I'm just going to give that one more playback almost 2023. OK, but it is definitely quite noticeable. It's not coming from the cable. It's coming from the microphone. So that is unfortunately why it's not a great interview mic. Now, just before we wrap up, not to not to end this on a negative note. So I put into the videography subreddit, which I highly, highly recommend if you are also learning about videography and you're looking to get advice from other people. I asked the Reddit community, well, what is a good XLR handhelds mic that is suitable for stuff like Vox Pop or or even this kind of a thing I was doing with Noga where you're interviewing somebody and you want to pass over the microphone. So in that kind of an application, handling noise like the one we heard is not acceptable, right? And that was me handling the microphone very carefully, right? If you're in the in a street and you're, you know, a bit less, you can imagine it would be very problematic. So a few recommendations from Reddit that are now going on to my never ending video wish list. So this is what they recommended for a sure interview mic. It's the Sure SM63L and you can just see how differently it's described for handheld application on stage broadcast, recording and TV studios. So at $180, you know, this one's quite a bit more expensive than the PGA48, but they're explicitly saying it's, you know, workable for broadcasting, so you're not going to get handling noise on a microphone. So if we just come back to the listing for the PGA48, you can see it very specifically says it's a handheld mic for vocals. And again, that's where the problem comes using it as an interview mic. A couple more suggestions and I'm going to have to have a think about which one I want to invest in next. I'm going to use my PGA48, but probably only for video podcasting, right? Where each microphone was sitting in a stand and therefore that handling noise wasn't a problem. It's not the end of the world to edit it out. But you know, if you can, if you're looking for a good XLR wired interview mic, it's, you know, definitely worse spending $60 more to get something actually designed for this job that's not going to have this obvious flaw. So the second option here that Reddit recommended to me is the Audio Technica 80804L that comes in at 110. And finally, a classic, the Sennheiser MD46. And you can see here again, just to kind of emphasize the kind of nuances you might miss in product descriptions, if like me, you're new to buying microphones, high quality dynamic microphone for rough use in live recording and broadcast environments, especially tuned for interview situations, special design avoids wind and handling noise problems. So again, that's being very, very clear that it's intended for interview use. And it's not going to have any issues with handling noise. And that microphone from Sennheiser comes in at 200 at USD. And it is a XLR with a dynamic cardio cardio pick up pattern. So the summary is that while I think the Shure PGA48 is a relative bargain at $40, if you're looking to use it for interviews, vox pop broadcasting, field recording, the handling noise is problematic. It's worth spending between $60 more for something like that audio technica mic or $150 more for the Sennheiser MD46, which is a really solid choice and probably the microphone I'm going to pick up next time I am over in the USA. Hope that was useful. And if you do want to get more videos from me about videography, audio, other technical subjects, do consider subscribing to this YouTube channel.


Hello and welcome to the session. Let's discuss the following question. It says the total cost and the total revenue of a firm that produces and sells x units of its product daily are expressed as cx is 5x plus 350 and rx is 50x minus x square. Calculate the break even points and the number of units the firm will produce which will result in loss. So let's now move on to the solution. We are given that the cost function is 5x plus 350 and the revenue function is 50x minus x square. Now we have to find the break even points. Now at break even points cost function is equal to the revenue function. So we have 5x plus 350 is equal to 50x minus x square. So we have this implies 50x minus x square minus 5x minus 350 is equal to 0 and this implies minus x square plus 45x minus 350 is equal to 0. Now taking minus sign common we have x square minus 45x plus 350 is equal to 0. Now we'll factorize this quadratic equation. So we have x square minus 35x minus 10x plus 350 is equal to 0. Now taking x common from the first two terms we have x into x minus 35 taking minus 10 common from the last two terms we have x minus 10 into x minus 35 is equal to 0. Now taking x minus 35 common we have x minus 35 into x minus 10 is equal to 0. So this implies x minus 35 is equal to 0 or x minus 10 is equal to 0 and this implies x is equal to 35 or x is equal to 10. So these are the break even points and this is the answer to the first part of the question. Now in the second part we have to calculate the number of units the firm will produce which will result in loss. Now for loss the revenue function should be less than the cost function for loss rx must be less than cx. Now rx is 50x minus x square and this is less than the cost function which is 5x plus 350. So this implies 5x plus 350 plus x square minus 50x is greater than 0. This implies x square minus 45x plus 350 is greater than 0. Now this is the same quadratic equation we got in the first part and it gets factorized as x minus 10 into x minus 35 greater than 0. So this implies x minus 10 is greater than 0 and x minus 35 is greater than 0 or both are less than 0 that is both factors are less than 0 then their product will be greater than 0. So now we take two cases x minus 10 greater than 0 and x minus 35 greater than 0. So this implies x is greater than 10 and this implies x is greater than 35. So the company will be in loss if it produces the units greater than 35 since it says that x is greater than 10 x is greater than 35. So the common x greater than 35 now case 2 x minus 10 less than 0 x minus 35 less than 0. So x is less than 10 x is less than 35. So again the company will be in loss if it produces the units less than 10 less than 35. So the common x less than 10 x 4 or hence will be in loss 35. So this completes the question and the session. Bye for now. Take care. Have a good day.


Hi. Sorry, everyone, when you've got ears as big as mine, you feel a little bit self-conscious with these things. So my name's Cass Halcroft. I work at Bloomberg. I work on the cloud infrastructure team there. We run numerous OpenStack clusters at Bloomberg. The actual production code that we use, we'll look at it a little bit, is available on GitHub. If you go to github.com. You can see exactly what we're running in production, which are nuts. And we're going to talk a little bit today about how we went about upgrading all these live production clusters. So I've been asked this a couple of times, actually. We're sponsoring the DevLounge. If you want to see a Bloomberg terminal, you can go down there and see one in doing its thing. And I've been asked this a few times. So I just put a slide up in case you haven't heard of Bloomberg or a little bit unsure about what we do. If you go into almost any large financial institution, you'll see a bunch of very high octane people clustered around screens like this. You trade those and risk people and such and such. All poking and making trades and doing things. That's the Bloomberg terminal. That is our core product, or more properly, the Bloomberg professional service, which has been essentially pretty much any financial data you want and a lot of analysis. And if you go down there and have a look at what we have, it's really quite amazing. But in addition to that and supplementing that, we have a lot of other products. Bloomberg News is probably in most of your hotel rooms, at Bloomberg Business Week. We're also in a lot of other verticals, particularly law, sports, and government. So we're branching out into other areas. Just to give you an idea of the kind of numbers we're dealing with, we have something like 22 million instant messages a day, quarter of a billion, excuse me, messages a day.  I think that number is actually a little bit out of date now. And we deal with, what's that, 50 billion ticks a day, at least. So what is the Bloomberg cluster private cloud, or BCPC for short? It's part of a much bigger cultural shift at Bloomberg towards a DevOps culture. We all know what this is to reduce our developer. The friction for product deployment to reduce our turnaround to market, all these good things like machine provisioning as code, we're an essential part of that. We're the private cloud component. And when we were designing this, we had some very specific design goals in mind. We wanted to have a complete automated install of an entire cloud stack, not just OpenStack, but everything, soup to nuts, right the way from provisioning the boxes, the actual hardware boxes, right the way through to monitoring and alerting and what have you. So if you go and look at GitHub, that's exactly what you get. You get everything we run, the entire stack. Each cluster in itself should be highly available, active active, highly available. And we wanted to make it as simple as possible. We don't want terribly complex interdependences that things can go wrong. So it's a simple design. We also want to try to keep it as homogeneous as possible. What I mean by that is there isn't lots of special dedicated nodes. We're running just MySQL or just Nova or something like that. We have one stack. And then we just slice that stack to add compute power. We'll have a look at that. And it was also designed, excuse me, to scale purely horizontal scaling from a single node up to many hundreds. So you can run BCC on one node on your laptop or three nodes on your laptop. Or you can run it in many hundreds, and it's exactly the same architecture. You don't change anything apart from just provision more nodes. And one thing that's often overlooked in environments such as ours is it has to be completely deployable in the absence of any internet whatsoever, so in a totally isolated environment. So we have recipes to get out to mirrors and get everything you need to install this thing in total isolation. So as I said, it's not just OpenStack. It's not just Seth, but OpenStack. It's a whole lot. And this is a list of the technologies we use. I think it's fairly exhaustive. Rather than go through them in great detail, you can just go and look at our code on GitHub and have a play with it. To sort of graphically represent what it looks like, and I want to go through this in too much detail, but just to get an idea of what we're trying to upgrade, we have, obviously, the hosts at the bottom on Seth. There's our storage layer that acts as both S3 storage and sandbox storage. We have MySQL Galera and Rabbit MQ cluster, and then we run the OpenStack services in a sort of shared nothing architecture, and we front that with HAProxy and Keep Alive D. I said that we have one stack. Obviously, we don't run hundreds of Nova schedulers. We have a reduced stack where we essentially just, for adding compute, where we just slice off the top layer of this and we just run Nova compute and Nova networking, and we keep everything else the same. We have a, well, obviously, we don't run MySQL there as well. The compute nodes not only contribute to Nova compute, but they also contribute their disks to the Seth pool. Yeah, so every hardware I know, if it has N disks, it'll have one root partition, which is the OS and then contributes N minus one to the Seth pools. Okay, so a little bit of a note about our development. We do a release cycle that sort of tries to sort of match OpenStack's release cycle. We actually call it leases slash ice house is our current production release. We're working on Juno, and that sort of comes out a few months, hopefully after each OpenStack release. The one thing to note is that you can actually run an entire BCPC stack on your laptop. This laptop's running one. You can just spin it up inside VMs. This is a three node. It's got one head node with the entire stack on it and then two compute nodes and a bootstrap node, which we'll talk about a little bit. So you can go to GitHub and just spin one of these up. So that's how we do the development. That's our stack, what we actually run in production. We have lots and lots of individual isolated clusters. So rather than have one large OpenStack cloud, we have lots and lots of smaller ones that live in different networks in our environment, but also of course in different data centers. So each cluster is, each network zone is mirrored in of course at least another data center. And by policy, we apply a policy where every app that runs on us at the platform layer has to be able to run on more than one data center. So it can fell over to the other side, no problem. So how do we actually go about deploying it? We have a deployment node, which we can push, it's just a bootstrap node, which we can push code into our Chef server. We can control Cobbler, which we use for provisioning hardware. And we can push new packages into our app to mirrors. And from there, we can also talk to our out-of-band management. So IPMI tools to talk to our out-of-band management. So how do we go about upgrading this? So our policy is that we'll only ever upgrade one network segment in one data center at a time. So we always ensure that at least that network segment is available on the other, at least another data center. And then we, because of our policy with our apps, it means that they can just fell over to that other data center and keep on running. But we do try and keep the downtime, any downtime to an absolute minimum. So our target downtime is roughly one hour to upgrade a cluster to have the control plane back up and running. One thing that's important to note is we have a lot of data on our Chef clusters, and we can't just like take that off and put it on tape every time we wanna do an upgrade. So we have to keep all our Chef data, you know, live on the cluster and keep it safe. We can't just sort of blow everything away and start again. That's not really an option for us. We replicate within a cluster three times. We do not replicate across the river or across to another data center anywhere else. So we have various options when trying to upgrade. We could try a rolling upgrade when we did Grizzly Havana. This wasn't really an option. So we didn't design our upgrade procedure around that because it wasn't really available to us. We could just do an app get upgrade, but we do so many architectural changes. We, you know, when we were upgrading from our version, our stack from Grizzly to our stack in Havana, we've changed a lot of other things, not just open stack. We've changed, you know, made improvements to our own architecture, made improvements to our self-crush maps, et cetera, et cetera. So upgrade isn't gonna get us there. So rather than doing a lot of tedious smucking around without get uninstalling stuff and reinstalling things and things like that, we just tend to wipe host and reinstall it. But we've got a lot, a lot of clusters to do. I should just add a curiosity. How many people here have been personally responsible for upgrading open stack clusters? Okay. When you're doing an individual cycle, how many independent clusters have you upgraded? Raise your hand if you've done more than two. Raise your hand if you've done more than five. Raise your hand if you've done more than 10. Okay. All right. So it has to be automated, right?  So we developed a workflow in Ansible. That runs through a cluster, one, two, three, to five. The number of nodes you actually do at any upgrade at any particular moment depends on, you know, what they're doing. Head nodes I'd probably do one at a time. Worker nodes I'm a bit more confident about. I'll do, you know, three at a time, depending on your Ceph crush map, if you feel comfortable doing a whole rack at a time, you do a whole rack at a time. One important thing is that you have to run a bunch of functional checks before you start to upgrade a host and after you're done. The possibility of cascading failures is really kind of scary. So if you've got all your checks in place and you've got all your motivation in place, you just sit back and relax and watch it roll through the cluster, right? Right? No. So that's the long periods of boredom. 2 a.m., sitting there, watching your Ansible scripts running, running, running, running, running. That's the long periods of boredom. A little note on state. So if you understand, of course, we're nuking individual hosts, but we're not gonna wipe out the OSDs. The OSDs stay where they are, they're separate disks. They live through the upgrade. And our cookbooks are written so that when you bring back the root partition, it will reincorporate those OSDs back into the Ceph cluster. So that all works rather nicely. So Ceph maintains our images, our volumes, our S3, all our OpenStack data. And then MySQL Galera, of course, is the OpenStack database. As I said, Ceph generally does a pretty good job of reincorporating those OSDs out of trouble.  So sort of detail the plan, what actually gets automated. We take down, oops, seems to have got squashed a bit, we take down the control plane, first of all, it's sort of time zero, so this is the action, this is the impact it has on our tenants and how long, you know, in the, how long it takes. So we take down the cluster, Vip, that's the front layer, that basically means that at that point, our tenants can no longer control their VMs. The VMs are still there, they're still running, and they're still absolutely fine, but they can't spin a new ones up, destroy them, et cetera. And we also, obviously, disable Chef at that point. After that, we actually upload all our new code into Chef, you know, what we're targeting. You deploy new app to mirrors if we need to, et cetera, et cetera. Then we upgrade Ceph, if required, not all of our upgrades require an upgrade in Ceph, but the grizzly, no one did, for example.  Upgrade MySQL, that usually goes pretty smoothly. So you're, of course, this thing is clustered, so you wanna make sure that you, when you bring up the new host, you're in the same version as the ones that are keeping the data for you. So you have to do that before you actually start to reimage the new head nodes.    And at that point, you've got your control plane back. You've got a control plane available to your tenants. Now you don't have any compute, unless, of course, Nova Networking and Nova Compute is backwards compatible. So the details of what actually happens on that last stage is you IPMI off your server, re-enable Pixie booting, chef in a new OpenStack control plane. All the recipes, like I say, reincorporate the OSDs. They rejoin MySQL cluster, they rejoin the Rabbit cluster, everything's great, and then you do your DB sync, and the world is a fabulous place. The compute nodes are a little bit easier. Once you've got the fully operational control plane, you don't have anything to control, but now you can start to rip through your compute nodes and upgrade those in exactly the same way. And, of course, they'll just rejoin the cluster. At that point, your tenants now have new compute nodes to deploy. Now, if they're running VMs and they haven't terminated their VMs, if they're running VMs on those compute nodes, at that point, they'll get shut down. But, of course, we're running all our stuff with boot-from-volume, so when the compute node comes back up, they can just actually basically take that volume and restart their VM if it needs to be. So, that's essentially what we're saying here.  We do tend to encourage people to terminate their VMs, it's just cleaner, and they can bring it up on the other side of the river, it's not, you know, for our apps, it's not a particularly big deal. So, we're chasing through our cluster, doing one head node at a time, doing three worker nodes at a time, and then before every iteration we move on, we've got to check that everything's okay, you know, so some examples as Ceph returned, as MySQL being fully synced, as Rabbit joined, you know, is there any errors on the functional checks, and then we can move forward. And at no point, we're risking any of our data. So, as I said, this all works great.  So, you have, at this point, you have a choice to make. You've got a node that has maybe a root volume that is not gonna come back, so you either bring that host down, down and out it from Ceph, let it migrate the data off, or you fix it in place and move on. You just gotta be a little bit aware of cascading failures that you've got maybe something wrong in your new recipes, and if you kept on going, you're gonna end up with, you know, a rack and a half of completely bought nodes, and now you've got some data problems. So, I generally stop when I hit the first or second error, and go back and just check that I'm not doing anything silly, and then make a decision about what I'm gonna do about that particular host, and it's case by case basis. There's always problems. The Ceph attack, so as I said, we're not just upgrading OpenStack, we're upgrading Ceph, we're also upgrading our architecture. So we make improvements to our crush maps, we make improvements to how we lay out the data on the disks, et cetera, et cetera. So that obviously has to get chefed in when we bring up a new head node, it's gonna apply those changes to the crush maps. So some things you have to be really careful of here. For example, when we went from dumpling to firefly, there's this rather innocuous little tunable that we just put straight into our crush maps, which it says right there in the documentation, it causes every single byte on your cluster to move, which is great when you've just done one head node, you've got a control plane back and no compute, and now your entire cluster is trying to rebalance. So you have to be quite careful on how you manage that. The way we do that now is we have switches in our recipes where you can basically say, you know what, I know that this crush map is not how you want it to be ultimately, but just leave it alone for now, and we'll come back and we'll do it later. Let's get everything back up and running and then we'll apply some of those crush map changes, those tunable changes later. The database nephew, so obviously you got to back up for your database, right, it's the first thing you do, but then you're doing 10 or so clusters, at some point you're gonna have a database drop when you're halfway through a big sync and now you have to blow it all away and reload the database and start again. So that's another thing that has gone wrong a few times. The deprecation warnings that you get, I don't know if you all got these when you go from Grizzly, Havana, those will come back and bite you in the butt later, so you better take care of those during that upgrade process. And there's also some BCPC related changes, like we changed from going from one large pool for block storage to SSDs and HDDs separated out in the G2H upgrade, so you have to go into the database and actually make some changes related to our own architecture. So that is essentially how we do it, we call it the rolling nuclear update. We just roll through the cluster, blow one away at a time, maintain the state of the data as we go. Because of our policy applied at the app player, we can have that short outage at a control plane level in any particular one data center, I let it fail over to one of the others. It's actually works really well and it works, I think it will continue to work, hopefully in the future we can have that backwards compatibility so that the VMs that are already up when we bring up the control plane will be able to still apply some to control cells and do some live migrations, let's hope. Doing it this way, we don't expose any of our data, we don't expose any of the CEP pools to any potential data loss as long as we do all those functional checks. And of course we just completely automated this because there's no way I'm doing this 10 times by hand. The only downside to this option is that you will take a short outage at your control plane level and of course every single VM would have to be restarted if you're in an environment where you don't have this fail over option, you would have to essentially go back and restart those VMs as you go through the cluster. So that's the downside. So that's it. Any questions?