Welcome back to our lecture series Math 1060, trigonometry for students at Southern Utah University. As usual, I'll be your professor today, Dr. Andrew Misildine. In lecture 19, we're going to talk about the half angle identities, which are related to the double angle identities we talked about in lecture 18. In fact, it's essentially the half angle identities are the double angle identities worked backward. What do we mean by that? Well, let's first take a look at them. In this video, we're going to talk about the half angle identities for sine and cosine. If you take sine of A over 2, where A is any angle, and you're considering half of that angle, so we go from like a 30-degree angle to a 15-degree angle, sine of A over 2 is going to equal plus or minus the square root of 1 minus cosine of A over 2, where this plus or minus, as we've seen before, is dependent upon the quadrant that we're in. Sine's going to be positive, the first and second quadrant will be negative in the third and fourth quadrant. Without knowing the quadrant, we can't predict the sine. We can get the absolute value here, so we will need some information about the quadrant to proceed forward. Similar for cosine here, cosine of A over 2 is going to equal plus or minus the square root of 1 plus cosine A over 2. You'll notice that the half angle identity for sine and cosine are almost identical. You get plus or minus the square root of 1 plus or minus cosine of A over 2. The only difference is in this sine right here.  And this goes with the usual analogy, we are not analogy, but the mnemonic device we had before, that cosine's a jerk, right? Cosine gets along with cosine, but cosine and sine, they kind of butt heads a little bit there, even though sine wants to get along with cosine. So you have a negative sign right there. So where does this, where do these interesting half angle identities come from? Let's consider where the half angle identity for sine comes from. To get this we're gonna consider actually the double angle identity for cosine. Like I said, the half angle identities come from turning these double angle identities on their heads. Now with the double angle identity for cosine, there actually was three different versions of it. Cosine of 2x could equal cosine squared x minus sine squared x. We're not gonna use that one right now. There's cosine of 2x equals 2 cosine squared x minus 1. So there's a version that only involves cosine. We'll use that when we do the cosine half angle. But for the sine half angle, we're gonna use the version where cosine of 2x equals 1 minus 2 sine squared of x right here. It only involves sine. And what we're gonna do is we're gonna solve for sine right here. So to start, we're gonna subtract one from both sides. Cosine of 2x minus 1 is equal to negative 2 sine squared of x. Now we're gonna divide both sides by negative 2. We get that sine squared of x is equal to, well in this situation, because we're trying to divide by negative, it's gonna switch to sine on top. So you get one minus cosine of 2x like so. And this sits above two. And then to solve for sine, excuse me, solve for sine here, you can take the square root of both sides. You'll get the square root of one minus cosine of 2x over two. And then again, because you're taking the square root, we don't know which sine is which. So we're gonna do plus or minus right here. So you're just gonna start with the observation that a equals 2x, therefore x equals a halves. And you make that substitution in right here, x is equal to a halves, and then 2x is equal to a. And so this gives us the half angle identity we saw right here, all right? Let's do the same thing for cosine. Again, it's the same basic calculation for cosine. You're gonna take cosine of 2x is equal to 2 cosine squared x minus one. So use the other double angle identity like we talked about. We wanna solve for just cosine of x right here. So subtract or add one to both sides, excuse me, cosine of 2x plus one equals two cosine squared of x. Divide both sides by two. You get cosine squared of x is equal to here. We're gonna get one plus cosine of 2x over two. And then take the square root, you get cosine of x is equal to plus or minus the square root of one plus cosine of 2x over two. And so using the same substitutions, you can replace the 2x with an a and you can replace the x with an a over two, thus giving us the half angle identities from before. Now, why did we go through all the details of proving these formulas here? Why not just memorize them? Well, one, it can be difficult to memorize these things by seeing where they come from. We have a much better chance of memorizing them and then in the horrible situation which we need them, but we can't remember them, we could actually reproduce them possibly on the fly. But another important observation here is that there are alternative versions of this. This one formula is not all there is. In fact, this equation right here is equivalent to all of these equations right here. For which, again, this one right here is just the double angle identity we saw before. So the two are really equivalent to each other. You know one, you know the other, but one is often more useful than the other. But also something we see a lot, like for example in calculus, some of the intermediate steps are extremely useful. For example, you take this one right here, the way it's usually written is sine square root of x is equal to one half one minus cosine of 2x. This identity, I can't emphasize enough, it's not the version of that, it's not the half angle identity version that we're taking as the official one. But this version right here is extremely useful in a calculus setting. It can turn a sine squared into some type of linear combination of cosine of 2x. Alternatively, this one right here, same basic idea, we can turn a cosine squared, let's see a cosine squared of x is equal to one half one plus cosine of 2x right here. This version of the half angle identity is the one used the most often in a calculus setting. So by going through the proof, we actually learn lots of identities all at once. So let's actually do some practical applications with this half angle identity. Let's consider the observation that cosine of A is equal to three fifths. And we also know that angle A is between 270 degrees and 360 degrees. And so let's then find sine of A halves, cosine of A halves and tangent of A halves. Now you might wonder, why did we say A sits between 270 degrees and 360 degrees? Why not just say that A belongs to the fourth quadrant? Well, this one does imply that, but the other direction is not necessarily true. A could be in the fourth quadrant, but its angle measured might not be inside of the domain 270 to 360. For example, if A was equal to negative 45 degrees, that's in the fourth quadrant, but there's a little bit more to it. And the reason this is significant has to do when you start considering half angles, right? Because if A was equal to say like negative 60 degrees, which again, which is in the fourth quadrant, if you take half of A, that's equal now to negative 30 degrees, which is still in the fourth quadrant. On the other hand, if we take something like A equals 300 degrees, which is in the fourth quadrant, if you take half of A, sorry, half of A, then you end up with 150 degrees, which is in the second quadrant, like so. So with half angles, knowing the quadrant is not good enough, we need to know, it's better to know the angle measure clearly. We won't always know the angle measure, but if you give us some more information, we can then figure things out from there. So that's why it's described in this manner right here, 270 to 360. All right, so the half angle identity we saw for sine, sine of A over two from the previous side was plus or minus the square root of one minus cosine right here of A over two. So some things to do is since we know cosine, you can do this calculation here. You're gonna get the square root of one minus three fifths all over two. Now, what about the sine? Well, this is what we were trying to think about earlier, right? If A is between 270 and 360, then what happens when we cut these things in half? We know that A over two will be less than half of 360, which is 180 degrees. And then A will be larger than half of 270, which is 135 degrees. In particular, A halves is gonna be in the second quadrant. That's what we need to know right here. Now, in the second quadrant, right? First quadrant, it's positive, positive. In the second quadrant, it's negative, positive. So sine is going to be positive in this situation. Cosine will be negative. We'll get to that in a moment. So how do we deal with these fractions instead of fractions? Not a big fan of these nested fractions here. Let's simplify. Let's times the top and bottom by five. This is gonna give us a square root of five minus three over 10, like so. Five minus three, of course, is A two. And so then we get two over 10. Oh, I should have followed my own advice. Now, what about the dominators? Cos two goes into the 10 five times. You're gonna get the square root of one fifth, which if you prefer, you can also write this as one over the square root of five. Or if you prefer, you can take the square root of five over five. These are all equivalent to each other. I think the simplest version here is actually gonna be one over the square of five. So I'll take that as sine of A halves. What about cosine of A halves? Right, using the half angle identity, we get plus or minus the square root of one plus cosine of A over two. So in this situation, we can plug in cosine of A, which is gonna be one plus three fifths all over two. Since we're in the second quadrant, we're gonna be negative in the situation. Again, let's times top and bottom by five to simplify that fraction. So we end up with negative the square root of five plus three over two times five. I learned my lesson that time. Five plus three is equal to eight. Eight goes into two, of course. I should say two goes into eight, four times. So we end up with negative the square root of four over five, for which four is a perfect square root of two. And then we're gonna square to five again. So that'll then be our cosine here. We get negative two over the square root of five. And so now we're in a situation where we can compute tangent. How are we gonna do tangent? Do we have a half angle identity for tangent? There is one, but honestly, since we already have sine and cosine, we're just gonna use that one, right? We don't need another one. And so we're just gonna take sine of A over two, which is one over the square root of five, which we saw. And then we're gonna take cosine of A over two, which is negative two over the square root of five. Again, fractions to the side of fractions. Since it's a fraction just by divided by a fraction, you can just think of it as multiplying by their reciprocal. You get one over the square root of five times square root of five over negative two. We see that the square roots of five will cancel out. And so tangent in this situation would be negative one half. And then we could figure out C-canco, C-canco tangent to A over two very easily with the values we have now collected. Let's do another example of this. This time we know sine of A is equal to negative 12 over 13. And we know that A lives between the angle measures 180 degrees and 270 degrees. So again, I'm telling you that A lives in the third quadrant, but with half angles, we need a little bit more information than that. So we're told it's 180 to 270. So if we just cut that out, cut that in half immediately, we get A halves will be larger than 90 degrees, which is half of 180. And we get that A halves is less than 135, which is half of 270. So this tells us the important information that A2 is gonna live inside the second quadrant again. Can we find the six trigonometric functions here? Well, to do sine of A halves and cosine of A halves, we need to know cosine, right? So let's first convert from sine of A to cosine of A. And to do that, I actually like to think of right triangle diagrams. Admittedly we're in the third quadrant, my picture. So I could draw this a little bit better, but that's okay. Just think of it in terms of reference angles or something like that. Here's A and like so, sine is opposite over adjacent. If we started off in the third quadrant, right? We have plus plus, we have minus plus, we have plus minus. And in the third quadrant, everything is negative. Well, sine and cosine are negative, I should say, like so. So if sine of A is equal to opposite over hypotenuse, we're gonna get a negative 12 right here and 13 right here. By the Pythagorean equation, the other side should have a length of five as we're in the third quadrant, excuse me. This length is gonna be a negative five. So we can then compute cosine. Cosine of A is equal to negative five over 13. We need to know that for the half angle identities, okay? So now let's do them sine of A halves. This is gonna equal in the second quadrant. A halves is in the second quadrant. Sine's positive there. We're gonna get the square root of one minus cosine, which is negative five over 13, all over two. So we have this fraction here. So let's times the top and bottom by 13. This is gonna give us the square root of 13 plus five all over two times 13. Let's see what happens there for a moment. 13 plus five is 18, like so. 18 is divisible by two, leaving a nine behind. Nine over 13, nine is a perfect square. So we're gonna end up with three over the square of 13. We'll describe that as sine of A halves. All right, let's use the half angle identity for cosine now, continuing on. Cosine of A halves, this is equal to, well, it's plus or minus. We're in the second quadrant for A halves. So cosine's gonna be negative right there. We get the square root of one plus cosine, but cosine is negative five thirteenths, like we observed, all over two. Let's clean up this fraction again times top and bottom by 13, giving us, of course, a negative, the square root of 13 minus five over two times 13. 13 take away five this time gives us an eight. So negative eight over two times 13. The two goes in eight four times. We get negative the square root of four over 13. Four is a perfect square. You end up with negative two on top and then we get the square root of 13 again. So this time, now that we have sine and cosine, we can find tangent, just like we did in the previous one, right? Tangent of A halves is equal to three over the square root of 13, that sine, times we're gonna get the square root of 13 over negative two. So we're divided by cosine. The other square root of 13 is taken away and we end up with negative three halves for tangent. If we wanna find cosecant of A halves, we just take the reciprocal, we get the square root of 13 over three. If you wanna find secant of A halves, you just take the reciprocal of cosine, so you get negative square root of 13 over two. And if you wanna find cotangent of A halves, we'll take the reciprocal of tangent and we end up with a negative two thirds like so. And so we can compute each and every one of these trigonometric ratios using the half angle identities. The critical thing is to find of course sine and cosine. Once you find sine and cosine, you can compute the other four trigonometric functions with ease as we've seen many times before.


Alright Mayor tell us what we're doing today. Well we're starting our first Wednesday rides. We'll be riding every Wednesday through October at 530 different locations throughout the city to try to promote cycling and fitness and awareness in the community. Healthy communities are much more vibrant and strong communities. And tell us that you have a big vision of part of its accessibility and accessible government. Tell us about how this fits in the Absolutely. You know I always say this is your Fort Worth and you need to tell us how you want it governed. And we did some of these last year and people would ride along with me and tell me about their neighborhood or tell me about their streets or their churches or what they saw from their council people and various and so many things. It's really interesting what people will tell you on the bike because they get warmed up and they get loose and you know you feel friendly and accessible and that's what we want. This is all about being real open and free with the citizens and letting them know we're here and listening to their issues. And these rides aren't for hardcore bikers. They're for everybody. Oh no these are for everybody. We won't drop anybody. Every pace can ride with us. The only thing we ask on this is that no kids on training wheels or anything. But other than that anybody can come ride along with us and we'll have a sweep at the back and somebody leading the backs and we'll kind of take turns rotating around. And tell us how they can how the citizens can find out about these rides. They're on Facebook, go to Mayor Betsy's Facebook page and also on the city website. And we'll do some weekend rides too right? We'll do some weekend rides. We'll do some weekend rides and we will do some kids rides along and we'll do those weekend rides in different sectors of the city in the neighborhoods and then we'll do some kids ride in some of the neighborhoods also. And tell us about the Trinity trails. I mean this we have a great system here don't we and some to promote with it. We do. We have fabulous trails. 44 miles of paved trails and a good many unpaved trails and it's a great access to walk, run, roller glide, just to get out with your family, ride your bike. Even people do need to be aware though that there are rapid riders and slow riders and dogs and kids and keep in mind that you really shouldn't have your headset on. You should be listening to the people passing. We want everybody to be safe on the trails and have a good time. And as mayor, why is it important to be active? It's important to be active because for one thing just flat makes you feel good. But it also helps just promote overall health and well-being. Plus it gets you out to see this beautiful city we live in. Okay, thank you so much.


Senator Smart, a DME, has again called on the federal government to fast-track the completion of the Ajau Kuta steel project, which is a catalyst for industrialization of the nation's economy. A DME also urged the federal government to use the current gains made from crude oil sales at the international market to fix the comatose refineries or build new ones. He lamented the abandoned Ajau Kuta steel complex, which he said has the potential of providing gainful employment for over 50,000 Nigerian youths, calling for the resuscitation of a full-fledged industrialization is to be achieved in the country. What is wrong with our nation? Today, we should be smiling that the crude oil, the barrel, is not selling above $100. At the time, I was afraid for Buwari's government because crude oil was selling at $10, $15 when this government came to power. Now, today it's over $100, but we're not getting the benefit. But why did you take this government to block the leakages? We must do it and use this advantage to complete ayuna steel complex. I was in Lagos yesterday, and I looked at all the steel that were used to build a new terminal. They were all imported. And you know what I think? We have them in Kogistate. For me, as an individual of Kogistate, I feel pain, and at times I feel like crying because if my government was allowed to run ayuna steel complex, we'll be sporting by today.


Welcome to the World Summit on the Information Society 2016 in Geneva, Switzerland. And I am delighted to be joined by Marta Tomowska, Minister of Information Society and Administration in Macedonia. Marta, welcome. So please do tell me what sort of progress is Macedonia making into becoming an information society? Thank you very much for your invitation. Macedonia, I'm really happy to say that it's making relatively good progress in terms of development of information society. And I say relatively good because we always can do better and can provide even more, let's say, services to our citizens that are electronically, electronic services, for example, for e-government services, e-business or whatever is needed. However, the situation is good not only because we are really good positioned on the indexes that international organizations are measuring. Success rate of the implementation of Internet Society like ITU index, like World Economic Forum or World Bank. But it is the real situation in the country if you come to Macedonia, for which I'm sending you the invitation now. You can see that there is infrastructure and there is easiness, let's say, to access the Internet from everywhere. You go even from the hotels, from the streets, from the cafes, from the offices whenever you are. So that makes us like one of the countries in the region that has good results in this area. We have 70% of penetration of broadband into the country, while about 40 to 45% is on a fiber optic. We have covered the entire country with mobile broadband. Both of our mobile operators have LTE networks, so it's 4G networks. You say both of the mobile operators. I imagine that for competitive pricing you would like more mobile operators to enter into the market to get prices down. That's exactly true. So we are planning to amend the regulation by opening the space for even more competition. So we would like to invite, let's say, mobile network operators plus mobile alternative operators to enter the market. So the end users, Macedonian citizens, have even better prices for the mobile services. One of the issues, of course, is making sure that everybody can have access to the Internet, whether it's the young or the elderly, and the elderly often feel that they've been left behind. What are you doing in Macedonia to make sure that they're part of the information society? Yes, since there is good infrastructure, we are also planning to have the broadband as a universal service. Therefore, we are launching a tender for an operator that will provide fixed broadband even to the remotest village in the country, which will be done by the end of 2017. The infrastructure is there. The motivation is something that we need to work on. We have started even to introduce the information technology, even from the elementary school to our kids, by launching a project, computer per child, even like seven or eight years ago. So the youngsters are ready to adopt the new technology. For the working people, they have Internet, of course, because of their job conditions. And for the elderly people, they are really motivated to use the Internet because we are always promoting the benefits from the Internet. They can be in touch with their relatives all around the world. They can read the news, get informed, learn something new that they didn't know. And we did two interesting projects in order to make the Internet and the entire technology accessible for them, which was implementation of 680 Wi-Fi kiosks all around the country, plus 25 Internet cafes that are sponsored by the government, where these people are getting basic Internet skills trainings. So in fact, Macedonia is an applicant country for the European Union. You're well on the way to meeting the targets of the digital agenda 2020. Exactly. We are chasing the same goals that are valid for European Union members. By this, we are planning to have broadband Internet with high speeds by the end of 2020, which means that the entire population has to have access to really fast broadband, which is about 30 Mbps, while half of them, 50% of the population should be connected via 100 Mbps. Marta Tomowska, Minister of Information Society and Administration in Macedonia. Thank you very much for joining us. It was my pleasure. Thank you. Thank you. And please do join us on the ITU YouTube channel for more interviews with ministers, regulators, business people, non-governmental organizations, who are all trying to make sure that information communication technologies are reaping the benefits for everybody around the world.


Well, um, can you see me right now? Am I still at 144p like my last video? Thanks, YouTube. I mean, I was trying to talk about this thing back here in New Merch, like sponsoring my own video, and wait a second, what's happening? What? Ah! Ah! Okay, seriously, I actually don't even know what this works at, as I'm recording. YouTube has actually had a number of issues going on. It's really weird, because when I private videos, it still looks like it's 4K60, but then I publish it and it's 144p, they're still processing and still checking my video from over a day ago. So whatever, we're just gonna move on and pretend there's nothing wrong, because I can't control YouTube, but I can control what we talk about on this channel. And I got a three-pack of news to kick off your morning today. Stuff that I'm pretty excited about. In fact, there's actually some really cool things in here about not just Game Boy and Game Boy Color, you know, coming to Nintendo Switch. We have an update on that story. It looks like other platforms are going to be coming to Switch as well. So that's really exciting. I mean, N64, GameCube anybody? Game Boy Advance, DS, where does the buck stop? I guess we'll have to see. We also have interesting conversation to talk about in terms of fanboyism and how a certain particular fellow content creator has been impacted by it. And then we got to talk about the Tokyo Game Show and what's happening there, because there's some big games being shown off that are coming to Nintendo Switch. And also, Nintendo's a publisher at Gamescom for like the first time ever. What does it mean? Let's find out together after I remind you we are giving away a Nintendo Switch OLED. That's right, you can win any white Switch OLED. All you have to do is be subscribed to the channel. We will announce the winner in a live stream in the beginning of October. Hope to see all of you there. Thank you so much for your support and let's get into the news. So Eurogamer put up a new article, essentially using their own sources to confirm that Game Boy and Game Boy Color games are coming to Nintendo Switch. We already had this story technically from Nintendo Life earlier this week and it actually started with the Nate the Hate podcast. So yeah, Game Boy and Game Boy Color games seem to be a lock to come to Switch at least according to all the rumors and reports. And it seems to be coming this month, likely announced in the Nintendo Direct that we are just patiently waiting, somewhat patiently waiting for Nintendo to announce. But what's more interesting here is Tom Phillips, one of the writers at Eurogamer, put out a tweet where he was obviously advertising the story and he mentioned that, hey, there are other platforms actually coming to Nintendo Switch online that haven't leaked yet and he's aware of. Essentially, bam, we're getting more than just Game Boy and Game Boy Color coming to Switch and he knows what those platforms are. They haven't leaked. So at this point, what is coming? Is it gonna be Nintendo 64, which felt like the natural progression after the SNES? Is it the Game Boy Advance that many people were hoping was going to come along with the Game Boy and Game Boy Color? Is it GameCube, the infamous system that's never been on any virtual console before? Are we talking Wii, really?  What about the Nintendo DS? I think obviously we can rule out Wii U and 3DS at this time, but yeah, I feel like the whole of Nintendo's gaming goodness and glory is open season. So we'll have to wait and see what actually ends up coming besides Game Boy and Game Boy Color, which again, is technically still just a rumor at this point, hasn't been confirmed by Nintendo, but it is something that we are really looking forward to here at Nintendo Prime and we'll just have to wait and see obviously what the game lineup is because I feel like the game lineup matters even more than the platform. The platform just gives us expectations on what the lineup might be, but obviously Game Boy and Game Boy Color is exciting, but N64, that's pretty hype. GameCube, that's pretty hype. DS, Game Boy Advance, all of that's pretty hype stuff. So we'll have to wait and see what's gonna be added. I presume those platforms will be added next year. They seem to be adding new platforms once per year kick. Maybe Nintendo's actually gonna ramp it up and add them more often. We did skip a year last year due to the pandemic, so maybe they actually have a third platform or really a second platform since they consider Game Boy and Game Boy Color the same platform. Maybe they do have another one ready to go and they're gonna announce it alongside the Game Boy and Game Boy Color or maybe they'll have it come out early next year. So next up, this is an interesting story and I feel kind of bad for a certain other YouTuber. His name is Austin Evans. You guys might be aware of who he is. He's actually one of my favorite. Do we call him a tech YouTuber or maybe a tech entertainment YouTuber?  He's done a number of videos. He was actually one of the first YouTubers to go hands-on breaking down and taking apart and putting together X-Wax Series X, which kind of leads to some of the current controversy. So for those that aren't aware, there is a brand new version of the PlayStation 5 that has come out in Australia and Japan. This version of the PlayStation 5 weighs about a half pound less than the launch model. And so Austin Evans actually got his hands on one and made a video titled, The New PlayStation 5 Is Worse and that's obviously his opinion. Now he clarifies in this video why he thinks it's worse and it basically turns out that while there is a slightly redesigned fan in the system, that's not why it weighs less. It actually uses more material so it would add to the weight. It's because it uses a significantly reduced heat sink. The heat sink is almost cut down in a third, maybe even a half of the size of the old heat sink on the PlayStation 5, likely as a cost-saving measure, which Sony tends to do with their systems over time, they do a lot of cost-saving measures. And in doing that, it obviously weighs a half pound less. Now he did do an outside vent heat registration thing with a non-calibrated heat sensor that you can buy for your phone and it showed consistently that it is actually throwing more heat out the back of the system, suggesting it's doing a worse job cooling the platform than the old PlayStation 5, about a five degree Celsius difference, which is not insignificant. Now we did note in this video that the performance of the platform didn't seem to be worse. You weren't getting worse frame rates. You weren't getting worse resolution. And he noted that this was likely just a cost-saving measure and they determined that the PlayStation 5 didn't need as big of a heat sink. It was probably over-engineered. So when he said the PlayStation 5 was worse, it's just his personal opinion that yeah, it has worse thermals, which makes it a worse system, but not necessarily worse in the ways that gamers are going to care about. And of course the PlayStation faithful couldn't just let a man's opinion be. There was a lot of attacking of him on the internet. People talking about how, oh, his heat instruments aren't calibrated and yada, yada, yada, which is fine. You can use that as a point of criticism, although I will note even if it's uncalibrated, as long as it's the same heat instrument being used to measure the heat out of the back of both systems, it's gonna be accurate for whatever that actual heat instrument is measured to.  Even if the overall temperatures aren't accurate, it'll be an accurate difference between them. But that's neither here nor there. You could talk about how, oh, it's not a true measurement of heat. That's just what's coming out the back of the system. He didn't put diodes like on the actual die, like, you know, like Gamer's Nexus and others do.  But for some reason, because Austin Evans is suggesting there's a new version that's costing Sony less money to make with a smaller heatsink, which is factually true, that outputs more heat out the back of the system, which is from what we can tell is factually true, that he is a shill, he is a shill for Microsoft, and that he should be doxxed. That's right, people went as far as to dox Austin Evans, leading to him making a follow-up video. So I've gotten into some controversy. This is not a video that I wanted to make. In fact, I've actually put off making this video for a few days because I was hoping that things would just kind of blow over, but it's just escalated, right? Like I've been doxxed, it's sort of crossing some lines that I'm just not comfortable with. I'm purely talking about the old PS5 versus the new PS5, they're charging you the exact same amount for and yet has a inferior cooling solution. Look, if you think I'm wrong, that's your opinion and be my guest.  Catch it on the controversy. Not like I've never done a video on something controversial, but things are, this is a PS5, right? The fact that the doxing and some of the more extreme elements of this, I think are way out of proportion, right? I get it, I am making content, I am making things that you may not agree with. I stand behind the video I made, I stand behind my opinion, which again, agree with or disagree with, that's fine. But the level of toxicity in the community I think is, it's a lot, right? Now, not to name names, and it's not, this is not me, but people I know in the sort of the space have gotten like death threats and just like ridiculous stuff, which is, there's no excuse for that. All of us as YouTubers get our fair criticism in return, but for us to make response videos to that criticism usually means something went beyond normal and getting doxed is just not acceptable. It's the same thing when someone's live streaming and you call in fake stuff on them when you find out where they live, like fake kidnapping.  The SWAT team is just doing their job because they take those threats very seriously. So it's not to the level of obviously it's life being in danger, but getting doxed is just wrong on all levels and just massively immature for a reaction to a personal opinion. You could have an issue with how he came to that personal opinion, but it's still just Austin Evans' personal opinion that the new PlayStation 5 is worse at least in the cooling department than the previous PlayStation 5 and he noted in his response video, like yeah, his opinion doesn't change, he's doubling down. Yeah, you can argue there's better ways to measure the heat and better ways to get to his conclusion, but the point is that he still feels the new PlayStation 5 is just factually worse than the old one and it's his personal opinion. I don't know why we can't just take people's opinions at face value and I've dealt with this as a YouTuber as well, not quite to the level of Austin Evans, I don't have an entire fan base trying to dox me, but I've dealt with the backlash of personal opinions and having informed personal opinions and having people not like those personal opinions. Actually, it just happened yesterday, there was a few angry commenters because I criticized the pricing strategy of the Crisis Remastered Trilogy on Nintendo Switch because we're paying $40 more than everybody else for digital files, it doesn't really make sense to me, but it is what it is and that's before you consider that it's a worse version, I'm not saying it should charge less than the other platforms just because its performance is lower, but at least the same, right? So I got attacked for having that criticism, which I felt was a pretty informed opinion and here's Austin Evans showing on video the reasoning he feels it's worse and even if you disagree with the methodology used to get to his conclusion, it's still just an opinion. There were even people attacking his thumbnail for crying out loud, don't know what the problem with the thumbnail was, he was absolutely correct that the original thumbnail that had just showed the actual PlayStation 5's heat sinks at a quick glance was kind of not that great of a looking thumbnail because people are scrolling, you know, you get about a half second to a full second maybe to grab people's attention with a thumbnail and it just didn't at a quick glance like look like anything at all kind of like a blob.  I mean, okay, yeah, he's obviously trying to get your click but is it baiting when in the video he shows you exactly why he feels the PlayStation 5 is worse? No, it's not bait. So that's a whole conundrum that I just wanted to cover because one, I fully support Austin Evans and any other content creators ability to post their personal opinions, especially when they provide evidence to back up that personal opinion, even if that evidence isn't exactly the 100% most accurate way to do things. And again, he doubled down in his video saying, hey, just because they did this doesn't mean that the PlayStation 5 performs worse in games. It also doesn't mean, and we can't even know this at this point, that the new PlayStation 5 won't last as long as the original PlayStation 5. We literally, the systems haven't met on the market to know if there's a longevity difference between the two. Typically we know throughout history with technology the hotter something runs, the worse it is at dissipating heat, the overall that usually doesn't last as long as something that's better at it but again, the heat difference could be negligible and both the components might die around the same time, say in 10 years or something. So I don't know, I just find the whole thing to just be a fascinating deep dive look and it makes me wonder, how does Nintendo and Sakurai deal with all the backlash over decisions that they make whether personal or otherwise. It's always fascinating to me to see fan bases overreact to things that ultimately don't really even impact them just because Austin Evans provides some evidence why he feels the PlayStation 5 is worse and you can disagree with how he got to it. That conclusion, it doesn't change you enjoying your PlayStation 5 and it doesn't mean if you buy a new one that's using the cheaper heat sink that suddenly you're getting a worse experience. You might get a slightly hotter room, but okay. I mean, I have a gaming PC in the studio and it can even heat this room up a lot so I don't think it's really a problem. All right, so moving on to our last story, we have the Tokyo Game Show. The Tokyo Game Show actually comes here in a few weeks and it happens every single year. It's basically Tokyo's version of E3. It's been going on for a long time and Nintendo has essentially never been part of it but there were some hopes out there for fans because this time around for one of, I think it's might even be the first time in Tokyo Game Show history, although I'll have to go back and look. There might have been another time. Nintendo is actually listed as a supporting publishing partner for Tokyo Game Show which got some people excited to think, man, Nintendo's actually gonna bring Metroid Dread and others to Tokyo Game Show. That's really cool of them. Nintendo's never really supported the show despite being the biggest video game publisher in Japan and being literally the most profitable and most high-valued company in Japan. I know there's Sony and others might have a higher ceiling with their stock and everything but Nintendo has more money and zero debt and in the bank and just liquidated assets. Their company has the most core value right now today. If you had to liquidate Sony, it wouldn't be worth as much as liquidating Nintendo. This came out a couple of years ago. Nintendo's just literally the richest company in Japan and that's really cool, even bigger than a lot of car manufacturers. But here's the deal. That's cool. Nintendo's going to be a publisher there but Nintendo, because they're a publisher there, their Japanese Twitter had to come out and clarify that yeah, we're not presenting any games. Sorry for any confusion. They are listed as a publisher because they are there to support some of their indie partners. If you didn't know, Nintendo does publish some indie games so it is, you could theoretically argue Nintendo is showing off games at the show, games that they are publishing but they're just supporting indie developer partners that wanted to be at Tokyo Game Show. That Nintendo might, as an example, let's say sports story is at this event. Well, Nintendo is paying and publishing that game but if the developer really wants to be at Tokyo Game Show Nintendo's not going to tell them, no you can't because you're on our platform. No, Nintendo's going to support them and try to push them at the show if that's where they want to be, same with any other indie partner they have. But setting all that aside because this is basically business as usual, Nintendo's not really going to be at Tokyo Game Show, just some of their indie developing partners that Nintendo's publishing. There actually is some great news for Switch owners at Tokyo Game Show because we're going to see basically new gameplay and trailers and all that for some big third party games coming to Switch. In particular from Square, Capcom right now we're kind of waiting and seeing if any of their stuff is going to become and I think there's a Monster Hunter Rise update that they're going to be showing off but Project Triangle Strategy is going to be there which is interesting because Nintendo's publishing that game in the West but Square's publishing it in Japan, hence why it's there. So Project Triangle Strategy is going to be there but really the default two is going to be there which I find interesting because it comes out on PC before the show, it's already out on Switch. In fact, it's almost sold a million units on Switch. That was a new story this week, so that's really cool and I got some more news for Bravely to follow so stay tuned in a moment here. But yeah, there's also going to be that Guardians of the Galaxy game shown off again which we all know is coming to Switch too. Now it's coming to Switch as a streaming game but it's still coming to Switch so it's still worth at least paying attention to what they're going to show and obviously we'll have to see how well the streaming version of it works when that game comes out. So that's basically from Square, I mean it's what we're aware of in terms of coming to Switch. We have to know yesterday we talked briefly, I'll link to that video even though it's 144p where we talked about THQ's situation here in a couple weeks as well where they're going to be showing off a bunch of games likely coming to Nintendo Switch. But getting back to Bravely to Fault, what's really cool is there was actually a new interview that dropped in Famitsu where the creator of Bravely to Fault 2 came out and said, yes, we're making more Bravely to Fault games. It's happening. Like we, he's basically confirming right now today they are developing Bravely to Fault 3. He didn't call it Bravely to Fault 3 but he basically means they're making the next Bravely to Fault game. And you know it's not some side spin-off game because he said, hey, yeah, we're going to make more Bravely to Fault games but just know it's going to be a while. He says it takes about three to four years to make a Bravely to Fault game. So, you know, we're looking at 2024, 2025 before we see Bravely to Fault 3 which could end up being a launch game for the next Nintendo platform or obviously a cross-platform game probably going to be on PC as well but launching first on Nintendo because obviously Square and Nintendo have a really tight partnership with that IP since it started on Nintendo platforms. So yeah, it's really exciting to see the future of Bravely to Fault at least being confirmed that yeah, Bravely to Fault 2 has sold almost a million units just on Switch. Yeah, we're going to make another one. Duh, we're going to make it in three to four years. Duh, we're going to take our time and we're going to do this right. I like that, I like that attitude and it's just good to know that this franchise isn't going to die or isn't going to stop at the second or isn't going to just go off in the spin-off territory and never get a third main entry. That's happening, that's great news. So now we could talk about the end of this video. All right, so look obviously, yesterday's video was a bit rough. I waited a long time to publish it because YouTube was having some issues and who knows what this one, I'll probably just publish this one right away. If it's in 144p again, I apologize. I know some people at like I actually have control over the final product that gets shown and I just want to clarify some things between, you know, from the viewer perspective to the content creator perspective because I used to be a lot like you guys where I get mad at YouTube when something didn't work as a user but then I'd also get mad at the content creator if they put out subpar content and by the way, maybe you think my video is just subpar in general, that's a general criticism of my style of video or my personality or, you know, I don't know how I pronounce things so people get upset about that because I do make some incorrect pronunciations. Yeah, that's how you say it, right? Or is it an annunciations? Whatever, I mess up sometimes. But what's interesting is, now that I'm a content creator and I've been doing this for four years, I mean, I've been on YouTube as a creator for longer than that but really the last four years was in Nintendo Primes really when I dove deep and cared about the actual people commenting on videos and all that. It's interesting because the only thing I really control when I upload is I control what the thumbnail is. I control what the description text is. I control the title of course so if there's any criticisms of the description, the title or the thumbnail, that's fine, whatever. The actual content in the video obviously as well because I made it. I control what quality I upload the videos at. Right now I do 4K 60 FPS but beyond all of that, I can control whether or not I want ads on the video. I don't necessarily control how many ads there are. I don't control the length of them. I don't control if they're skippable.  I have no control over what ads appear. That is strictly on YouTube side. I don't control what quality my video gets shown to you and I want that to be clear. The 144p isn't anything I can do about it. I can look at the video in private, like I can set it to private and look at the video and see what quality it appears YouTube has it at but what I see it at, doesn't necessarily mean that when I publish it, that's what you're going to see. Nintendo has this new check system which is checking your video for copyright content ahead of time. This check system is supposed to be helping out content creators by letting you know ahead of time if a piece of music or a trailer or something you put in your video is going to be copyright claimed and thus you're not going to be able to monetize your video and so it's supposed to help us as content creators know, okay, maybe we need to go back, make a quick edit to that one section, eliminate it or replace the music or replace that footage and then re-upload and go through the checks again. The problem is their check system broke yesterday and wasn't completing and then it wasn't also for some of us also not completing the processing and thus spitting out a 144p video that's still 144p at the time of recording and likely not been fixed. It's a really frustrating thing as a content creator because it makes it look like I don't care about the quality of content I'm pushing out to my end users and that sucks. You think I wanted to push out a 144p video where I have this logo and I'm advertising our new merch and all that? Like I want you to be able to see this. I want you to be able to see the merch. I want you to be able to see me. I wouldn't be on camera if that was the case. I just upload a still image with audio if I just wanted you to listen and I'll actually watch. So I care. It's just sometimes it's out of my control. A lot of things are out of my control. This does not mean I'm not thankful for the YouTube platform. Unlike many people, I actually remember the internet before YouTube and how hard it was to host video content, how expensive it is. Like we had a video one time back on Zeldinformer, a trailer for a new Zelda game. I think at the time it was, let me see. This would have been maybe a Skyward Sword and we weren't on YouTube yet. So we hosted our own video with our own custom video player that we had through an advertising partner who was providing it to us but we still self-hosted the video clip. That video clip went on to get like two million views which by the way, would be like the highest viewed video on this YouTube channel. The problem is that all two million of those views were streamed off of our actual server and it ended up costing us over 10 grand that month just for that video in hosting fees for the download. So I could appreciate how hard it is to do what YouTube does, how expensive it is and the platform they provide. I just want the platform to get better and I want it to be more transparent and bugs like this are just unfortunately part of the game. We're just gonna have to deal with this. Hopefully this video actually looks really, really great and you guys are getting a wonderful experience. I'll try to do the best I can to make sure that's what it's gonna be for you guys but honestly I don't have any control over that. Hopefully you enjoyed the video in general though and what I did provide. I enjoyed this honest conversation about what it's like to be a YouTuber because frankly it's actually pretty cool.  I enjoy the platform. I wouldn't be streaming and pushing guys to join memberships but probably the best way to support our channel. If you do want to throw us a couple bucks is to just become a member. I would like to get us over 100. I think we're at 77 out of 100 members right now where you become a member of our board of directors and that intended tribe had different tiers. So you hit that join button for that. Otherwise like the video, subscribe, comment. Those are great ways to support as well and help spread the love. You guys are amazing and I thank you so much for sticking by on this extra log video due to the end segment. Bye.


Hi, everybody. Welcome to our talk about how the CKI team, which I'm part of, is using spot instances. So the title was made at the time when I thought I'm going to give an introduction in many ways of how we would launch spot instances. But then after time, when I actually started to prepare the slides, it turned out that there are not so many ways to actually launch spot instances correctly. So the title might be slightly misleading. But maybe it's interesting, right? So I'm going to talk a bit about why the kernel CI environment actually uses spot instances, what it looks like when they fail in various interesting ways. Then because we just didn't know anything really about them, some of the things we learned while investigating the underlying issues. And then what we did to improve things so that they don't fail, right? So let's start with an introduction. So I'm part of the Continuous Kernel Integration Team, which is a team that provides pipelines, CI pipelines, as a service to the internal and external kernel development community. So the high level goal is that we try to prevent bugs from actually entering the upstream kernel by testing as early as possible. But a lot of our time is actually spent testing during the real development process. So nowadays, kernel developers sit on gitlab.com, like the internal ones, not the upstream ones. But the internal kernel developers, they sit at gitlab.com, they file merge requests, and they get pipelines. And CKI is responsible for keeping track of the system and making sure it runs. So I linked the project documentation. All our code is on gitlab.com. It's open. So you can take a look. You can send merge requests if you want to contribute anything. But yeah. So our main job that we figured after a while is actually that we run this service. And the two things we mainly do is we make sure that we can build the kernel or that the kernel for these merge requests is built. And we use spot instance for that. So that's what the talk is going to be about. And then we also test the kernel or these merge requests, or also the RPMs that are finally getting built depending on what changes are there, trying to gate stuff so that we don't break composers, those kind of things. And I know there's some improvement there, especially around breaking composers and stuff. More than that. Yeah. So for the building part, what this breaks down to is that we built all the four supported internal architectures like x86, ARM, PowerPC, and S390x. And that comes down to about 300 hours a day of building pipelines that are spent building draft versions in merge requests that then also are getting handed to QE for testing, those kind of things. And because this is the kernel, these are pretty beefy instances. So they have 16 CPUs, 32 gigs of RAM, a local SSD for all the temporary storage during the RPM build. And these things are not cheap. Like these things cost in general about $0.70 an hour, like US dollar cents. US cents doesn't matter too much. So we try to optimize that. And that's why we use spot instance in this case. And at the bottom, you see a plot of the hours. So you see this nice pattern of the last couple of months where we can see that kernel developers are actually doing stuff like proposed merge request. And the workflow is in a place where they actually really use that. Like before, they would send a patch to a mailing list. They would be required to build it locally. They would send a scratch build to Boo to get it built. But nowadays, it's basically git push and boom, there comes a pipeline along. And this is what the pipeline looks like. So this is the general CKI pipeline, which after on the left in the prepare stage, you get the usual thing you would expect. Like there's some merging going on where the merge request is merged into the target branch. So you get like a merge result pipeline. Then it's getting built, which is the interesting part I'm going to talk about. We are publishing repo files. So QE can actually test these things. And then we also tested ourselves. So there's some automated testing going on in B-curve, which is a couple of hours on all of these architectures, making short boots, LTP. If it's a storage change, there's a storage test running, those kind of things. So there is automated testing. And it could all be better, but it's also not bad. And the way this actually works for the building part is that we use native GitLab CI pipelines. So I'm not sure how many of you used GitLab CI? Couple? Yeah. How many used GitHub actions? About the same. OK, we are getting there. So the way this works on the GitLab side of things is that there's a job, which is basically an expanded JavaScript. And it gets handed to a piece that's called GitLab Runner, which is a service running on some machine. And what this thing normally does is spin up Docker containers somewhere, but let's say on the host. And then it executes the JavaScript on this container. And that's basically it, right? There's magic involved, but this is basically it. But now, if you spin local Docker containers, you are constrained by the host. So if you want to have dynamic scaling by using something like OpenStack or Hyperscaler, there's a piece involved that's called Docker Machine, which is an interesting piece of software that's abandoned upstream. So it was developed by the Docker folks, which creates a VM on a Hyperscaler, basically, just calls the API that I don't know whether any of you were on the Kube world talk before, but they gave a nice list what this involves. It's not a lot. And then it installs Docker on this newly created machine. And then it forwards the Docker port of the Docker on this VM to the local machine where GitLab Runner runs. And then it basically talks to the remote Docker daemon along the socket. So that's all there is. And there's basically no difference between a Docker container running locally or a Docker container running on this newly created VM. So this is a pretty, I find, a pretty simple, nicely working system. Now, it was not the direction that the Docker folks wanted to take the whole thing. So that's basically why it's abandoned. GitLab forked it to be able to give it updates. And if you drill down into the Golang source code that it is at the end, there's basically one API call, which is called RequestSpotInstances. And the way this is normally configured is that you give it an instance type and you give it a data center where you want to have your machine created. That's basically, like you say, give me this machine in this data center and boom, there it comes. And that's all there is. And to say, before we used AWS for SpotInstances, we used OpenShift cluster. So Red Hat has some interesting, huge OpenShift clusters internally that have huge nodes, PSI, OpenShift. I don't know. People might know it. And they have nodes of 128 cores. So you can actually spin these containers on an OpenShift cluster as well. But a cluster is in some way static.  They normally don't scale down. And if it's a bare metal cluster, I mean, these nodes, they are bare or they are not, right? So switching to something like AWS provides for a more, far more efficient environment where you only get these machines when you need them. Like if American current developers wake up, they will do it. And at night, nobody's running any of those pipelines normally. So the question was, are we using cost compilation or native compilation? It depends. So we are doing native compiles on ARM64 and x86, nowadays on AWS. So we're using the native machines there. For S390X and PowerPC, we are mostly cross-compiling them.  So there are some interesting aspects to BPF or whatever. Yeah, some pieces don't like to be cross-compiled. Officially, the official RPMs built on Boo, they are always built natively. But for our CI pipelines, where it's really the point of actually providing results fast that are good enough. And there have been discussions where this breaks down. But nearly all the cross-compilers work, they are supported by the compiler team. So this is actually a supported workflow for this case. But for the official RPMs, they are natively compiled. So this works nicely. And we were really proud of ourselves. We moved off the OpenShift clusters that were failing once in a while that had scalability issues. And we moved to AWS and it's stable and it's a hyperscaler. And it's all dandy, right? Like that was two years ago. And it worked nicely until the end of last year. Right before the holiday season, beginning of December, our jobs started to fail. And so this is what it looks like when a GitLab job fails. So normally, there's stuff up here. So it starts at line 104. So there's further things up there that actually work. And at a certain moment, if you get a message and it tells you, I cannot connect to the Docker daemon running on this VM. And the reason it can't connect to it is that this VM shuts down, right? Like, if it shuts down, there's no Docker daemon. So, yeah, basically it fails. And now these are jobs that take half an hour.  And if you do this in parallel, that's actually what we're going for. And especially for the kernel developers, that's affecting their workflow, right? Like, it's, these failures are annoying at best. And they're not just annoying, but they're really, actually really breaking the process. So this is first of December, somewhere around noon, Americans tend to wake up around the time, do a git push or something on their merge request. And then it just explodes, right? Like, these are within the time of 15 minutes, like huge loads of jobs just failing. And we iteratively, or we continuously restart them on an exponential scale so they get restarted directly and then after 10 minutes and then after 30 minutes. And so basically it's already on the 10 minute mark. So it tried a couple of times already and it just can't get anything done. So this is what started to happen beginning of December, like when everybody goes on PDO already. And basically it says 30% of the jobs failing on a daily basis, but actually what it meant was that during the American workday, nothing would actually compile. And so we started to dig down into it. And if you look at the logs on the runners, you find two error messages. And one is the first one, which is unfulfillable capacity, where AWS helpfully tells you that there's not enough capacity available to match your request. This is what you get if you do a spot request. Doesn't matter what you put in as a prize, basically, I mean, first they take them away from you and then they prevent you from spinning more, right? Like, so this is the first thing you get and I don't know what the sun knows them for the image builder team, but this is what you normally see when spot instances are not available anymore. But then we thought, I have spot instances, just let's just throw money at it. It's Christmas, holiday season, we have some cloud budget left, and we switched them to on-demand instances, which are the normal ones. And then helpfully, AWS came back with a different error message. In this case, it was insufficient instance capacity, and it was also the first time we've seen that. And they are actually more specific. They tell you the instance type you asked for is not available in this data center availability zone, and then it lies to you because it tells you that they are working on increasing capacity. They are not, right? It's just, no, obviously not, right? So that was not helpful. And then we kind of panicked and we started to switch instance types around and we kind of recovered a bit in panic mode. And that was the moment when we kind of figured that we just didn't know anything about spot instances and how this all worked, right? Because this was not supposed to happen, right? It's a hyperscaler, right? You ask, you get, right? Like it's capitalism, right? That's how it works. So money in some direction, and then supply follows, right? Like that's just how the world works on this side of the world, at least, right? So what is this actually? Like what is the cloud? What's the hyperscaler? Now, if you ask Wikipedia, it tells you that a hyperscaler or hyperscaling is the ability to scale with demand. Now, obviously that's a lie, right? Like because it did not, right? Like a hyperscaler did not hyperscale because we asked, we had demand, but they were not able to deliver. And then going to the specifics of AWS Cloud, looking into various sources on the internet, you find something that they have like about 30 regions, let's say cities, something like that. And then they have data centers, availability zones in their most availability zone is a data center. And each of these data centers has a couple of servers, right? Like above 50,000 per data center and these are, I think, big hyperscales. I'm not sure, yeah. David Duncan is not here, but I've seen him.  So that's my interpretation, right? Like I haven't talked to anybody on the AWS side. So what is AWS EC2 then? That's AWS renting you compute. So you can go there and say like, I want to have a server and they give it to you. And you, depending on how much money you want to spend, you can get different guarantees from them. So you rent the machine on the go, pay per minute, that's on demand. You can get a spot instance, that's what we are using. These are cheaper for various reasons. The main reason is that AWS can take them away from you. Again, when it decides for whatever reasons that it needs to. You can also go down to savings plans and reserved instances where you commit to use a certain amount of compute or a certain amount of compute per instance type. So that gives AWS some way of actually figuring out how much demand there normally is going to be. And we felt good about ourselves until this happened because we followed the recommended strategy of saying like, well, it's steady workloads. We do reserved instances, savings plans, those kind of things. Interruptible workloads, spot instances for all the rest, bursty stuff on demand. So that's what we did, right? Like that didn't help now. So we read a lot of stuff and we still didn't know what is actually going on. Now, on the word for the interruptible, that was something new that we didn't know is actually that AWS tells you a bit more about what it actually means. So you get a spot instance and it can be interrupted. You get two minutes, which is good enough for shutdown, but not much else. But they actually also tell you what the chance is that your instance gets terminated. So there's a helpful table and there's an API call and you can actually ask them how likely, if I ask for this instance type, is it that you will take it away from me within a couple of hours or whatever. And normally this is below 5%, so basically there's enough. But it goes up to 30 and further. So if you ask for the wrong stuff, you will actually have a very high chance that stuff terminates within five minutes, 10 minutes, a minute, it might actually just be really impossible to use as a spot instance. And then there's a bit, the problem of the whole thing. So there are real data centers somewhere, right, like this is not the cloud, this is a data center and it has servers and it's cooling as power. And these instance families are real machines, they're real hyperscalers sitting somewhere and they have a certain number of them, right? Like they don't scale, there's no scaling of a rack, right, like it's just there or it's not. And to give you an instance, when you request one, it needs to be there. If you want to request a non-demand instance, it needs to be there in this data center. And they can only give it to you if it's not used by somebody else. And this spot instance is kind of like buffer this range of instances that are available, but nobody's using yet, right? Like so basically you can use them until somebody wants them. But obviously there comes this moment when there's just nothing there anymore and then even on demand requests fail. And now, actually spot instances change in price and this is the plot at the bottom. This is for the last three months. I was too late with making my slides so I don't have any plots of the time when this all happened, right? But you can see at the bottom you have the black line which is the cost of some instance, pretty cheap. And you can see how the spot instances which are the colored lines at the bottom actually go up over time. Different availability zones, different data centers and they all kind of like follow the same trajectory. So somebody was asking for spot instances, they got more expensive. But then what we did actually was not in any way taking that into account, right? Like we just asked for this one specific instance type in this one specific data center. So there was actually nothing AWS could have done to help us, right? We just asked for the server and there was none. And this is basically what these messages mean. It means we don't have these machines available and yeah, okay, then a hyperscaler is also just bound to reality. So we were thinking about like what to do, right? Like we are engineers, what do we do if we have a problem? We add another script, another tooling, whatever. Our idea was to find out how can we specify multiple instance types? How can we specify multiple data centers? Can we maybe pull these APIs and figure out what to do about it? Maybe we can have something automatic that switches it around and stuff. And then somebody not known to the world of engineering suggested we should actually read the documentation, right? Crazy people, right? Like that's just not what happens normally, right? If you don't just read documentation. But yeah, okay, we did read, yeah. So we read the documentation and there are about five different ways to create instances in AWS, right? Like historical reasons, most likely, right? Like five different ones, right? And they are in the table. On the left, the most well-known is run instances and we've seen it in theCUBE, we've talked before. That's the one you find all over the place, right? It's the one you, it's named after run instances. Perfect, it's not recommended, right? Obviously not. And then there are two right below that have spot in their name. They give you spot instances. Run instances also can give you spot instances. But the other two that have spot in their name, they also, they give you spot instances. You're also not supposed to use them, right? Like AWS tells you, don't use them. Just, they are named like that, but you should not use them. What they tell you to use are the two at the bottom and one is called create fleet. So this sounds like, I don't know, ocean warfare or something, right? Like, I mean, wait, okay. And the other one is auto-scaling, create auto-scaling coop.  But that's the other one, right? Like they say, take these two. Okay. Which we did, obviously not, right? So that we figured we were, it was our problem, we knew that before. And so in the middle, reading a bit of the documentation about them, looking into what we actually wanted to do, which was like specify multiple instance types, specify multiple data senders. The three at the bottom actually give that to you. So they allow you to specify this in the API call. So you don't need any custom tooling. It's a sad world, right? Like you can't write code, you just need to change this API code. But okay. So the two rows still look good. And if you look like into the purpose we have, we want to spin one machine temporarily and then shut it down again. They are, it depends a bit how hard they are to use. And so create fleet is actually the one you want to use. You might not know, right? Like we did not. The internet also doesn't know. I don't know what chat GPT would tell you. So I mean, somebody can ask, it would be something if you're abroad, right? Like ask chat GPT what it would recommend to create an instance on AWS. Would be interesting, right? Like maybe it has catch on, but maybe not. So we used requests for instances. We should have used create fleet. That's what we would want to use. There are a couple of differences. I would just mention them. One is that request spot instances is simple. You give it something you want to launch, four CPUs, 16 gigs of RAM. It will give it to you. And it can also use something that's called a launch template, which is like a template where you can specify something that you can reuse and kind of like not have to specify it over and over again. That's basically all there is to request spot instances and create fleet has a couple of more cool features. One of them is that you can vary this launch template. You can say like, maybe do it in data center A, B or C. It's your call AWS. You can be more generic for instance requirements. You don't have to say like, give me this specific instance type like this magic number that nobody can decode. But you can actually tell it, give me something with four CPUs or more. And it will do that. And the last part is the allocation strategy where you can tell AWS by looking for an instance, please look that this instance is cheap and available, highly available, something that will not get killed. So create fleet is better, right? Like AWS right, most of them, they're right, most of the time so. And it's not hard, right? Like it's actually not hard. The sad thing about the whole story is that it's actually simple to fix. So if you want to create a launch template, which we did not, you specify this YAML file converted to JSON and feed it into the API call at the bottom. So there's an image ID, you give it the name, the image you want to boot. Secure shell key and then you say like, do we have here 16 CPUs, 32 gigs of RAM? Please don't use your really, really old instances that don't boot on relative anymore. And put a local SSD on those things. And they will give it, it's perfect, right? Like it's great, creates a template. And now if you look into it, there are actually 11 different instance types available, right? Like we asked for one, but AWS says, we have 11 of those, right? Like just, right? Just interesting, right? Like we didn't know, no. Yeah, anyway. And so the second part is that you now need to create this instance using create fleet. So you say like, I want to have an instance, instant instance, like now, please. Should be spot instance and spawn it into one of these subnets that are available. And that's it, why do you do this? You get an instance ID, it all works, right? It is that simple. So this was kind of like amazing in some way. Now the problem was, yeah, we didn't read the documentation, obviously. We still don't read the documentation, but we know a bit more about the AWS API and they do stuff to it, so like stuff like these allocation strategies, this is pretty new. Use create fleet, I can just implore you to do this or use order scaling, order scaling groups. This price capacity optimized strategy is basically what you want to use, like something that will not get killed and is cheap. Problem is nobody uses create fleet now, right? Like it's not something that is very common yet because everybody gets told use run instances. And so the thing we use, like Gitlaprano uses Docker machine does not support this because it's abandoned upstream. So what we did in the weeks towards the holidays last year was basically implement support for that. We forked it, I learned Golang, which was interesting. We put this in there and since then we've never had any troubles anymore with spot instances. So there's a happy ending to the story of CKI against the failing spot instances, right? So it all worked out. It was interesting. It was an interesting exploration and I thought that was a good reason to share it with you. So that's it. Do we have any questions? Yeah, so the questions, did it result in cost savings actually using this new API call? Not really that much because these things are mostly similarly expensive. But yet it was a bit cheaper than retrying the same jobs over and over again and failing over and over again. So we were just wasting a lot of money trying to get stuff done but eventually not actually getting it done. So the instances we might have gotten now they might actually be more expensive but because we only run jobs once in total it's I think similarly expensive as before but far more reliable. Yeah, so the question is are you able to restrict regions? Yeah, so this API calls are very specific to regions normally so you actually say in this region and then in one of the data centers but this is within a city basically. So this is like in the Amazon world a region is let's say a city and then availability zone is something like a data center. Yeah, so the question is is it possible to say like I don't care? No, it's not possible to say that and you also don't want to because one of the most expensive things in AWS is data transfer between various things. They are mostly free if you stay within a region especially S3, if it's all in one region it's all good but as soon as you move one of those out it gets really expensive. Yeah. How does this work because beakers go in and out? Yeah, so the question is how does this interact then eventually with Beaker where we are going to test these corners? So we are building them in AWS we are pushing them mostly to S3 buckets in AWS and then because transfer is expensive transferring it into the internet of Reddit is expensive we'd only do this once we copy them into an internal S3 bucket and then Beaker boots of a compose something internal adds the repository with these custom kernels, installs it, boots it checks that it's actually the one that's booted and then the testing progresses. Yeah, so yeah, kernels are just normal RPMs this is just a default repo and we tell it test this kernel and we make sure that it's actually also booted because that's kind of interesting. Yeah. So the question is if we try to actually limit the amount of jobs that it would spawn yes, we tried, but event like very, very fast it was actually the case that it would just not spawn any instances. So it was a single data center that was used by others because our load did not really increase over these time periods. Some other pesky customers came in and took our instances. So actually we were on the spot where we could not spawn any of these machines and our theory is that because we require these local SSDs and we think that there's a rack and a couple of SSDs that can be attached and if somebody else wants them I think there are far less SSDs than VMs that you can put on there. So if there's no SSD there might be compute but there's no SSD so you don't get any instance. So the question is did we actually observe any like intermittent patterns of if there's somebody else needs spot instances? Not really so we did not really figure out what was actually going on at the time but something unrelated is that in the last couple of months since about March spot instances have gotten far more expensive than before they settled around 30, 40% of the on-demand price and now in a lot of regions in a lot of data centers the prices have actually approached the on-demand instances people have looked into this and it corresponds with a higher rate of interruption so it's not just Amazon being greedy which might be something you would think but it seems like somebody or maybe because of cost savings people move more to spot instances so there's more demand for spot instances and so kind of like Amazon I think the algorithm in there tries to provide a signal for people to know oh don't use those types because we don't have enough of those and I think there's something happening in the background but if you don't use create fleet you don't react to the price indicator I think AWS wants you to actually use these API calls so they can distribute load across data centers across instance types but nobody uses the API calls so it doesn't work for them or not good. Yeah so the question is did we actually look into multiple regions? Yes we thought about it so there's we thought about multiple availability zones now these our like the networks we spawned some of them have internet connections that they are limited to certain data centers at the moment so that's one of the things we would want to enable spawn across multiple data centers spawning across multiple regions is interesting because as I said data transfer costs actually factor in quite a bit and so we would need to have S3 buckets in these other regions there's a transit gateway into the Reddit internet that would need to be provided there and also we have an open shift cluster that does a lot of the magic behind also what goes into the data and this one also runs in the same region so around data transfer costs this design kind of makes you not care which is bad in some way because it's cheap right like these jobs pull gigabytes of cash files from S3 it's free right like it's a very efficient way it's cloud native whatever but if you want to split this across regions or move to another region it would get prohibitively expensive so it's we thought about it but we didn't see any way yet to do this so yeah so the final answer so the question is like they've seen the same problem I don't know what team you are on but so other people have seen it at the same moment in time in US East 1 which is like the most popular one right we are not planning to move to other regions so you're free to move there I know that Reddit has internet connections in other regions as well but I mean if you have better experiences there maybe we'll follow you maybe but yeah it's all still right like it's okay okay so yeah that's one of those okay so the remark is that actually it might not be related to the SSDs because they haven't used SSDs on their instances but the problem was very much the same okay thank you very much