I'm gonna talk about the life of a CVE and understanding the project's release cycle. They're two very intertwined things. I'm James Strung from ChainGuard. That's me. Solutions Architect at ChainGuard, which just means I help customers secure their software supply chains, using sick store policy control or a bunch of other really cool tools.  I'm also a maintainer. I probably just copied and pasted this from other talks, so obviously I'm a maintainer. I'm talking at the maintainer track. I'm also an author of Networking in Kubernetes, a Klai Guru instructor on the same one, and a Gemli Cosplay enthusiast. I'm on the Community Alliance's team, so the mandate of that team is basically to work with folks like James and the rest of the community. All of my job folks is on open source projects and open source contributors. I'm my primary partnership for the architect side is the Ranger project. I also co-host the engine room with Alessandro here. That'll be, that might be on later today. I do a couple of things, game development. I also do cameras for something where we fly spaceships for a eSports organization called Admiral eSports, and I am very curious to see what James is doing now with someone. I should have wore it. I know. Sorry, agenda. We're gonna do a CVE review. We're gonna go through an actual CVE, the timeline, everything that happens behind the scenes. One of the things I wanted to talk about with this is just show you all that there are lots of other things that we're doing besides not approving your PRs. So, engine X core CVE remediation. We're actually gonna see what happens when engine X gets a CVE, which probably is a little more impactful, I would say, than Ingress getting one. And then we're gonna go through the release process. We can go in depth as much time as we have so you guys can see what happens when I break your clusters. But first we're gonna talk about a little bit about how engine X is configured, Ingress engine X is configured. So we've got about 118 annotations that we support and we're actually been working through and going through and making sure that they all have documentation, they all have end-to-end tests. And a new thing that you're gonna see coming out very soon is validations. So you see that from a annotation perspective, this is a configuration snippet, so being able to update the engine X config and controlling it through the annotation. So there's two ways to do that. It's the annotation and then a configuration option. So if you wanna make changes globally to engine X, you can do that with a config map. So you wanna set up things like the proxy connect timeout. So if it's not covered in an annotation, you can make sure it's covered in a configuration option. So that's how we kind of get the best of both worlds to be able to configure those. So annotations you can do per Ingress, the configuration options are globally for all of them. So that's how we get through that. But that sets us up for the CV that we're gonna talk about. So one, make sure that most folks, if you're not using it, you know how it's being used. So let's talk about what happens when we get a CVE and just walk through that whole process. So we do this in concert with the Kubernetes Security Response Committee. I think that's what the SRC stands for. Thank you, Carlos. He's not in his head. So the CVE that we're gonna talk about, we got the report last year in July. So it's a Hacker One report. Somebody submitted it to the Kubernetes Hacker Report, Hacker One team. And this is what the report looked like. So there's a lot on here on this page, but the piece that was important that we weren't aware of that could happen is that right here, somebody could mount the access token and you could request that. So that's the access token that has access to all of your TLS secrets on your Kubernetes, on all of your ingresses. So that was a problem. So somebody reported that and we went through and walked through how we fix that. So it took, it takes some time, right? The SRC is also a volunteer basis. So they got it, they verified it. They know that it was a reproducible vulnerability in August and they let us know around September. They probably emailed us sooner and I just didn't see it because I get 200 GitHub notifications and everything goes to a label called ingress. We got a nice email from CJ. He's great. He's with us on the SRC team. There's about five, four or five members. I've got a link in here that says who is all on the SRC and their whole processes. So Ricardo put together a remediation. Ricardo's one of the other maintainers. He's not able to be here with us. He's here with us in spirit.  So a lot of the times when we put PRs in and we accidentally break things or we're trying to put new things out there, it possibly is because we're trying to fix a CD that we haven't released yet. So October 29th, they released a CD. We have the fix in place. We've notified the distributors of that and we'll let you all know. From a CD reporting perspective, again we talked about the SRC team. Those are the folks who are working behind the scenes that when a report gets emailed in from the reporting. So these are all links to all of this to understand a little bit more in depth how that works. That was a very quick overview. And then we have that all in the NGINX security reporting to be able to report a vulnerability. But very quick high level overview. There's a lot of back and forth conversations about is this an actual vulnerability even after they've validated it and working through does the fix actually fix what was reported? With that, that was a very quick overview. But I wanna talk about, I saw this the other day when I was reading, I think it was about securing software builds.  That was a perfectly valid configuration option for a snippet. Like there's absolutely nothing wrong with that. So one of the things from the validator's perspective is we're gonna try to look at understanding what folks are trying to serve up. So obviously nobody should be serving up the service account token. There are other things that we're trying to remediate that like I said, there's 168 configuration options and all of that. So trying to make sure that a user is doing the right thing is very difficult because we allow folks to run arbitrary code. You can inject Lua code in there. It's just, it's really hard to do. So allowing people to run arbitrary code, it's part of it's gonna be on the ingress administrator, part of it's also on us. And it's just something that we have to continue to work on. So when I saw that, I was like, it's set up properly to do that, but there are some protections that we need to put in place to help work through that. So, and then the other thing, I lost was right. One of the things that we don't do and one of the reasons why we're putting the validators in place is that we don't validate user input. So how many people know about the emission webhook that we also use? It just validates the ingress configuration to make sure that it's a valid ingress object before it applies it because you have to do a reload and lots of things in that. So we wanna make sure it's a valid ingress object, but we don't actually, we're just looking at, does this actually work? But we're not looking at what's inside that. We're not validating what the user is trying to do. So making sure that they're not trying to mount the token or doing other nefarious things that they shouldn't be talking across namespaces, looking at things they don't have access to. So we're trying to work on validating more of that user input. So from an annotations and a configuration map perspective, that's why we're putting that in place for the validator. That should drop pretty soon. I think the PRs out there are just working through all of the end-to-end tests because we're doing this from scratch, 168 things. We've already seen that there's end-to-end tests and documentation missing on these things. So it's actually gonna be really helpful, I think, from that perspective. And with that, we'll go ahead and do the, we'll go ahead and talk with the CV remediation. All right, so I'm gonna cover Nginx core remediation and at a really high level. So I sat with the core PM 904 site. He said it was fine to give you guys your email address, but instead, I'm gonna just recommend you kind of come on to the GitHub repo, hit up James or I or get on the Slack channel and can raise and go through me and then I'll get whatever you want. Any questions you might have for Nina in this regard. But basically, the way our process works is we'll get a submission from something like a researcher and they'll come to us with the CVE and we'll take the information that they have and then we validate it by putting through it putting it through a set of like, you know, tests we all get in a room. We kind of argue over what is and what is correct what the threat vectors might be. Some of the things that also happened during this process is we first were just we're trying to decide is this or is this not a vulnerability? It obviously is almost always a vulnerability, but is it mainstream or not mainstream? So the big differential there is that sometimes we'll get vulnerabilities that are that are very, the use case is very small. So they're super, they're edge cases. They're very limited. So some of those we just kind of say, okay, this is more academic in nature. So we kind of put those in the, this is nice to know, but we're not going to do anything to fix those. But in other instances, the ones that make it to public we'll find out that those are legit vulnerabilities. Those get published. We work with the researcher to establish, you know how we want to do this. Then the next process is the information about that vulnerability goes into what's called an embargo. And basically the embargo is, is we work out with the researcher and the reporter what types of information can we relay? What can we send out to the public? What do we basically need to keep under wraps so that ultimately when this does hit prime time the researcher who makes their bread and butter off of finding and writing about these things can get the credit for the discovery process. There's no real money that, there's no money that exchanges hands. It's all basically a credibility based industry. So the next part of the process is essentially we've qualified it for mainstream. Its feasibility is deemed an actual threat. And then we basically go through the process of looking at further edge cases. So in some cases we'll get a vulnerability to come as one vulnerability. And then after we've done a bunch of research on it we'll figure out, well, this is, there's actually three vulnerabilities. So we'll basically publish additional CVEs to cover the additional threat vectors that occur there. This is also the point at which we kind of sort of decide how are we gonna fix this? What is that release process gonna look like when we do fix it? And how do we wanna go about the next steps? That process in total generally cut takes it roughly around two weeks. So it can be anywhere from three to seven days of just debating about the stuff that we just talked about. And then there'll be a part of that week where we're kind of figuring out how we wanna do this. And then there's like two weeks of actual code. Generally we'll get something out. So this is when the next step of the process we start working on the bits to publish the remediation which quite often comes in the form of a patch. And when I'm talking about core I probably should have qualified what that means. That means Nginx proper. So you go to Nginx.org you got the list of builds that are up there. That's Nginx core. So while it is the upstream or the progenitor for what happens in the ingress sig it also is used in by all of the distros for building distro packaging and a whole bunch of other things. So when we get a CV it not only affects ingress it affects literally the 1.7 million use cases out there that use Nginx. So we're incredibly tight-lipped and careful about how we fix things because obviously one minor mistake when you have 20 million websites that all of a sudden are not losing money so we really don't wanna do that. So once that's decided it's around two weeks of actual code and then by the 30th day we're usually looking at cutting a patch. That'll go out and during that time we'll notify the reporter. We'll lift the embargo. Folks like James will get information actually right around that time is when anybody who is an interested party is able to get the information. Now when I say get the information we don't have like a proactive notification that goes out with the exception of the list of PGP encrypted email members that are a part of the original search submission. Now this is something that we've kind of looked at as how can we make this how can we shortcut that process to make things faster for James and the ingress sig as well as distros in their packaging and on and on and on because obviously the quicker we can get that up to the universe anybody who's done the fair share of coding in here understands that busy polling is a wholly inefficient way to do anything. So proactively notifying people when that's available that's something we're kind of working on the details on so that's something we could look forward to in the future. And then 90 days is kind of this industry standard almost never do we go beyond the 90 day but this is what's considered the standardized patch cycle. A lot of this is around enterprise grade stuff because most people will only roll patches on a 90 day cycle. So some of this pertains to the plus product and products that are enterprise grade but most often we'll hit a release window of about 30 days. And that's pretty much the entire process. The next step in the process is it's handed off to the community and then James is the ingress sig as well as other groups that are interested parties kind of take it from there. I'm doing real good on time so if you have questions while we go through this release process just ask. We got 15 minutes still so I know we've got time for questions and things like that. So as I talk about this if you don't know what the release process is or if there's something that's like what is he talking about just throw it out there. So the ingress release ingress engine X release process we have four stages of it and each one of them is a different PR and I've been talking and thinking about how we can change some of these because as I break things I've learning new things about like there's a lot of get ops folks out there that will automatically roll out a patch release so that's fun to know and if I accidentally release a breaking change on a patch because I didn't know breaking change was in there I'll break things. So just learning and understanding that need to be very aware of things that could go wrong maybe not as a huge of an impact is engine X but still you can break a lot of clusters at once. You like breaking things so. Yeah well you know that's how I learn about it. Nobody tells me they're using ingress engine X so I only know when people are using it when it's broken. It was the best way to figure out who's using whatever rack thing you've got going on there is to pull the network cord and find a head way for the emails rolling. Exactly. So one of the first steps right is that we we manage this with a tag file. The tag file gets updated that's the one seven oh that will kick off a Google cloud build. So as a Kubernetes sub project we build everything in the in the Kubernetes realm. So when you hear about that that G cloud build I don't know what percentage of ingress engine X is responsible for that but I know it takes a while to build ingress it takes us about I think you'll see it in one of the slides. It takes about if there's an engine X change or if there's a CD remediation we've got to put out takes about four hours because we recompile engine X from source. So that build could take four hours if it succeeds. So is anybody using S390X by the way? No, oh that's going on the deprecation list. I don't see any hands. So we build across five or six different architectures across four or five Kubernetes versions. And then a lot of other things that I'll talk about in the later slide but that all gets built in this first PR. So that gives us the container image for the new engine X either the engine X or the controller. So all of our end to end tests everything that we're running we build on top of an engine X base image. So if there's a change or vulnerability in engine X or any of the pieces that we're using that's to container build. So I have to build the engine X container. I have to update all of our testing framework to use that new engine X build and then I can release a new controller. So I didn't even talk about that piece. That's a that's a pre PR for all of that. So that's kind of how things are built across that. So once I've got that new container build Shaw since also we're a Kubernetes sub project we also have to do image promotion. So the image promotion takes that from staging into the production. That's another PR for the Kubernetes org. How many people knew that about those staging about the promotions? Carlos don't raise your hand. You know about that. You write the code for it shaking your head. So in order for us to get that image out to you all we have to do what's called the promotion. So it's another PR another piece of YAML. So that takes anywhere from one minute to three or four hours depending on if I have one of my maintainers available to accept that PR. So that does the push that does the proud build that signs the container that gets everything ready. That's the point where the the folks who do the patch releases the get ops folks if it's available they'll run it and I haven't even done a release yet. So that's part of the release part that we're thinking about maybe changing of doing the GitHub release and then doing this this staging promotion. That way it's not available for folks to try to use but I'm wondering how many how fast some of you are are going to yell at me that that image isn't available when I do a GitHub release. Or how fast is your automation and picking it up? So we're working on trying to figure out how to optimize the build on trying not to break things or making things available until we've actually officially announced them. Excuse me. This is the one that recently has been automated. So all of those pieces are pieces of documentation. So updating the helm chart, we have to update the home chart version. The home chart has its own versioning strategy because a configuration option for a helm value could change but the controller we're not updating. So we can release the helm chart separately and independently but with a new controller version we have to do a version rev of the helm change of the helm chart. It also requires updating the change log as well. So helm, it's got its own release cycle. It's got its own change log but also we have to update it as well. And the version controller update the change log update we've been working really hard on automating that as well. It was an artisanal note that took me about three or four hours to handcraft and try to understand. But that's mostly automated now. And then also the documentation updates but actually thinking about getting rid of that last piece that documentation update because really all it is is changing the version string in the examples to the current version. I'm just thinking about just putting it to main so I don't have to update that one. So just working through some of those thoughts. So that's the one that's completely automated now from that perspective. But again, that's the third PR in this whole process. And then we've got that PR. We've got everything in. I tag the release and I make an actual official GitHub release. That's the end part of the cycle of saying that there is a new release. Here it is. But like I said, it's been available already for possibly a day. And there's no release notes. So when things break or something happens like guys, we haven't even released this yet. So can we not, can we wait? So we've got to work on that timing, the communication. Part of that is utilizing the Kubernetes Dev mailing list more. We actually have a very old Twitter handle that I'm getting the username and password for because KubeCon is great for tracking people down in person who's not answering your slack DMs. I know that because some of you do that to me. So I do that to others. So yeah, it's great to talk to people in person. So incidentally, that's one more person than we currently have to work on the SIG right now. So, and I don't know who in this photo is who. Do I get to be bomber? Sure. Okay, thanks. I don't know who I'd be. I'm just gonna be awkwardly standing up there. I use all my energy at Kubeoke or apologize that all of you. But that is the high level of the change release process. And again, like I said, it's three GitHub PRs and a GitHub release. Only the one piece is automated right now. So that could take anywhere from 30 minutes if we're on a Zoom call together and we want to get it out very quickly to maybe taking a week because people aren't available, people aren't on vacation. We only have three maintainers who are gonna approve that. One of them is in China and the other one also has a day job as well. So we're working with SIG Contributor X. I did a couple of workshops this week trying to understand how we can help build a community so we can take people who are one time contributors to getting them to a reviewer to ultimately getting them to approvers so that we can help grow and get those to be a little bit faster. We know that there is a backlog of PRs and issues. One of the ways that you can make sure that you can get your PR pushed through or you get your issue talked about is coming to the community meeting. Come to the community meeting, things get talked about, things get worked on and they get pushed through. That's how we got the open telemetry module updated. Is anybody using the new open telemetry module? Did anybody know that you could use open telemetry now with Ingress Engine X? Thank you, one person for validating me. How many people didn't know you could? Don't do that to me. So part of that is on the maintainers as well is getting that information out there. So trying to work on understanding how we can get more information out to the community. We have the two Slack, we have the two Slack's channels. So we're just working on what's the best way to get this information out to folks. I already talked about that, but just want to click through this so folks can understand. I presented this at the last KeepCon, I just want to keep reiterating this to folks. There's a lot that goes into Ingress Engine X, that's just not Engine X or the configuration option. So if they're with Golang, Alpine, Engine X and Lua, if there's a patch that's needed in any one of those or if there is, when we do an Alpine update or a Golang update, the Golang update is the same. We have to update all of our test runners so that we can actually compile it against that Golang version. So that's two more PRs, one to update the image and one to update the SHA for all of the testing. Same thing with Alpine, Engine X and with Lua. So did anybody know that we have a QCTL plugin? Yeah, we got two, two, two, one's better than two, so get in there. So the QCTL plugin, that's also something that we have to maintain. We've worked on automating that as well and we fixed it. The whole Mac switch and architectures helped fix that and shed light that we weren't actually releasing new QCTL plugins. So again, just learning new things. I think I've been helping maintain the project for, oh wow, two, three years now and I'm still learning new things about it. So trying to lower that learning curve for new folks. Two monitoring frameworks, that's gonna go down to one, once we deprecate those and push everyone to using the open telemetry module. So not going to do that until everyone knows that the open telemetry module is available. So thank you for that data point. Third party plugins, so from a Lua perspective, if you need a Lua module or if you need, I forget, I don't know the plugins off the top of my head. So we help maintain three of those. These are four hour build time. If there's an engine X change, there's just a lot that goes in. So the, any changes in those engine X modules that we use or someone requests a new one. And I'm not even gonna talk about that number, the depths, command line flags. So there's just a lot with the 400 end-to-end tests. Our end-to-end tests across those Kubernetes versions and now with LTS we're gonna have to do N minus five. It takes about 50 minutes to run an end-to-end test. And that's gonna be on every PR. So it's not a small process. If you have any questions, please let us know. This is the QR code for the feedback. I'll let us know how it went. We've got the links this, I haven't uploaded this yet, but I'll tweet this out. I'll put it in the Slack user channel, the Slack dev channel and try to get it out there. But we have a dev channel where we talk about the release process, PRs that we wanna work on, things that the maintainers need to work through. And then we have the user support. How many people ask questions and user support knows that it exists? Okay. I'm not getting a lot of hands up. How many of you use Ingress Engine X? Okay. So I'm glad, I'm really glad that it's, I'm sad, but I'm glad that we're having this conversation about all of this stuff and how this stuff works, because I figured that was the case. A lot of people use it, but they don't understand how the sausage is made. So when you guys go looking for new features and updates to Ingress Engine X, where do you look? Just in the release notes, okay. Well, that's good to know. We have a new contributor doc, but I think it needs work and it always needs work. Our docs always need work. We meet every Thursday at 11 Eastern. We've been trying to think about and move about the times, but it's hard because it's the middle of the day and everyone's working. And if you wanna talk, if you wanna see about what we do talk about and the meetings, community meeting playlists is out there as well. Right on time, I guess. Any questions? If you have a question about a PR, just no. I'm kidding, I'm kidding. Yes. So the question was, I mentioned the four hour build time. Does that include tests as well? No. That's compiling, so I'll go back. That's, when I say four hours, because we use Lure modules and a bunch of Engine X modules, we have to compile Engine X from source. We can't just use the Engine X binary. So that's the four hour build time. And that's across the, how many? The four architectures. And that's why I ask the question, because usually when S390 fails, it fails the whole build and that's really annoying. That's why I ask this S390 question. So yeah, it's not the testing. So we build that and then we do our integration testing with that new one. So that's why, build a new Engine X image. We do a pull request to update our end-to-end testing to use that new Engine X version and we can do the testing. So it's three PRs before we can even start testing it. We're working on optimizing that, but again, it's optimizing the release process, communication, or lack thereof, and also working through the PRs and managing all the other things. So yeah, it's fun. So that's why I made the joke before we started recording and asking folks that we need contributors to help and I need to do, we need to do a better job as the maintainers to lower that bar very eventually so we can get people to approvers. And Dylan, this one's for you. So, yeah. Thank you, there you go. But, anyway. By the way, that's a real axe, 100%. Yeah, I couldn't bring it. And wouldn't you say you can kill people with it or something like that? No. No, I would not. Anyway, thank you all for coming. I don't, were we 35? I can't remember. Anyway, any more questions? Let me know. We're hanging out at the Changar booth and at the Engine X booth.


Let's get into the big topic of open source, something that we actually have in mind. This is so awesome. We are an open culture that is actually already set up. What is that process that they've developed, or let's say... How's the Kubernetes ecosystem really going down? To this week's Ask an OpenShift Administrator Office Hours livestream. As always, that's a mouthful, and I am extremely happy to be joined by my co-host, Johnny. Happy Wednesday. How's it going, man? Happy Wednesday to you. It's been a good week. It's a great day at Red Hat, right? Every day is a great day at Red Hat. That's right. Drink the Kool-Aid. It's been a while since I was at New Hire, right? New Hire is where we get all the Kool-Aid. That's right. Yeah, I did mine six years this summer. That's crazy. Yeah, yeah. I was interviewing somebody yesterday, and they asked me how long I'd been at Red Hat, and I had to think about it. It's been three and a half years now. It's kind of crazy. Time flies. Time flies when you're having fun. Although, I tell you what, it's my wife, who is an amazing woman for many reasons, not the least of which she chooses for some reason to put up with me. She's a fitness instructor, right? And I decided I was going to go to one of her courses. She does one of those. It's like Orange Theory or one of those. I have felt every moment of every day since I went there Monday. And it's pain. I don't know how. And I didn't know this when I went there. It's one of those, like, all women's gym things.  It hurts to move my shoulder. Yes. That's awesome. So, yeah, I have felt every single moment. Yeah. Pure Bar. Yeah, Stephanie. Again, similar concept to that. Those women are, they're impressive. I'll leave it at that. All right. Well, so for everybody joining us today, thank you so much. We do appreciate you, you, well, viewing us. So this is one of the office hours live streams here on Red Hat live streaming, which means that we are here to answer any and all of your questions. So if you ever had a professor or manager or anybody like that to, you know, had an office hour where you could come in and just talk about whatever you wanted to talk about. That's what we're here for. So you are welcome to send any of those questions on whatever platform you happen to be watching us on, whether it's YouTube, whether it's Twitch, whatever that is, we'll get all of those chat messages, a restream here in the backgrounds, you know, it cross posts them to everything so we get to see all of them. And we're happy to address those as you ask them. And we'll do our best to answer them here on the stream. If we don't have an answer for you, then we'll take that offline and we'll follow up with whoever we need to on product management or maybe our peers on our teams, engineering as the case may be and we'll get that all taken care of. So I was watching the level up hour earlier and somebody came in at like the last minute and asked a question like as they were going off air. So hopefully they saw my message to send me a message, but I guess we'll see. So in the absence of any questions, and again, please don't let today's topic change any questions that you have in mind. But in the absence of any questions, we always have a theme or a topic that we talk about with each one of our streams. And today that is OpenShift 4.10. And in particular, we're going to be focusing on some things that Johnny and I feel are important to you all, to our administrator audience. So if you didn't see the what's new in OpenShift 4.10 live stream, so that happened two weeks ago. Yeah, two weeks ago. That was broadcast here on Red Hat Live Streaming. You can see all of that stuff. There's a what's new and what and next link. Let me dig up that link next. And maybe we have a handy shortcut. I'll post that on Twitch over here. So you can click that link that I just posted in there that'll take you over to a landing page that has both the what's new and the what's next. So what's new would be what's happening in the next release. So 4.10 in this case. What's next would be the roadmap. So the roadmap is going to be looking out more like six to 12 months and all the things that are happening there. And they do usually have an abbreviated roadmap slide in the what's new, but they don't focus on it too much. So yeah, Khalid, very much to your points. It was a dense session. They always have a tremendous amount of information that's in those. We've actually been talking about, you know, whether or not it would make sense to maybe reorganize how we do that presentation a little bit. We do it currently based off of kind of features and I think it's Conway's law, right, where communication happens based on the organization layout. So you can, if you follow along, you can kind of see how the teams are broken out.  I'm an administrator is maybe we should do it by personas. So, you know, have, you know, developer centric things and administrator centric things and, you know, that type of stuff and do it that way. So we'll figure it out. You know, hopefully it'll come out for the better. So thank you, Stephanie for posting that link in there as well. All right. So let's see, we've got just a ton of stuff to get through.  Cause yeah, there's just four dot 10 is packed full of stuff for anybody who Khalid, as you mentioned, right? Who watched that? What's new? So let's, let's talk about it. All right. So top of mind topics, a couple of things here. So one, if you are not familiar with the assisted installer, we've done a couple of live streams on it. It's a quick and easy way to get a new cluster spun up outside of your environment. So let me share my screen. And I want to share this window. All right. So what we're looking at here, this is a console.redhead.com slash open shift. So this is just where we go and deploy new open shift clusters and all that other fun stuff. So I'm going to go to create cluster here, select data center, and then I have this assisted installer. So assisted installer itself, as you'll see here is technology preview, but it is, which means that it's unsupported. But it's still a great way and it works really, really well. I, their success ratio is like super high or success percentage for deployments is super high. But it's a great way to kind of quickly get started deploying a cluster. And the reason why I'm bringing it up today is because they've made some pretty significant changes and updates inside of here. So I'm going to just create a quick cluster here. And one thing to note here, you can select RC versions of open shift to deploy. So if you want to try out open shift 4.10, which to be clear is not yet released, the bits are not available in a GA status. They are only RC. So if I wanted to deploy a 4.10 cluster here with RC six, I can choose that if I want to do a single node open shift. Remember that single node open shift is, or if you were deploying a GA version of open shift, it would only be supported when deployed natively to physical servers. But as you all have seen on the stream, you can deploy it to VMs. It'll probably work. It's just not a supported deployment yet. But you'll notice some other things here. So down here we've got like this enable encryption of installation desks for both control planes and compute nodes. And if I click next here, the biggest reason why I wanted to highlight this to you all is they have added some other things here. We've got this integrate with vSphere. So when I click the ad host here and I go through the process and I download the ISO, I can then provision my virtual machine, my VMware virtual machine. I boot it using that ISO and then it'll register itself down here in the host list. If it detects that those are VMware virtual machines, then this option will become available. And what that does is it deploys the cloud provider. So historically, Assisted in Solar would only deploy a non-integrated platform-agnostic deployment, but now they're starting to integrate with other platforms. So I thought that was really cool. You know, just another way that we can go through and get all of the stuff that we need, you know, quickly and easily for installation. And I think, I don't see it here, which is strange. I have an option here for deploying OpenShift virtualization out of the box too, but they do have OCS or ODF rather. And when you do that, it offers you the option of which host you want to use for ODF and all that other stuff, which devices do you want to use for ODF? So really nice method for deploying clusters if you have the chance. And that's an awesome upgrade too. I'm sorry I didn't mean to cut you off there, because I think when a lot of people do the single node, they don't realize, okay, I have to actually configure the local storage operator and then patch the registry so that way it'll start working. There's a lot of those things I think that are just kind of like ambiguous and that's awesome. And so I mentioned that the Assisted Installer itself is Tech Preview. The clusters it deploys are fully supported. Right? So if you, in the engineering team that's behind this, they're super good about helping with those things. So if you encounter an issue, open a BZ and more than likely they'll get to it pretty quickly. They're really awesome about helping out folks. And Andrew, we have a question from Rockhound1 about, is the Assisted Installer supported and disconnected or air-gapped environments? So, no. There are some Red Hat folks who have figured out how to do that. So, Johnny, I'm sure you've seen on some of the internal chat rooms where folks will say, hey, you know, here's my process for doing that. Ken is one of those folks. But officially, no. That being said, like further out on their roadmap, I know that that's something that they're looking at and they're looking at some doing some really cool things there. So, if you're interested, send me a message. And internally, I can share some resources with you for what they've got planned. And then there's another question from Fahad about the node tuning operator. Is if you make a custom change, is that supported through an upgrade? So if he goes and he changes the tuning operator and then he does an upgrade, is that supported as well? As far as I know, yes. So is it better to do Potter node label selectors? That one, I don't know that there is a, it's a subjective thing rather than a objective thing. So yeah, as far as I know, new tuning operator through upgrades is safe to do. That being said, I would say pay attention to the release notes because periodically, we will rebase CoreOS to new REL versions. So right now I think it's REL 8.5 based, 8.48.5. And it'll have in the release notes when it changes which one it's based on. So if you're concerned about any issues, or if you're setting anything in, you know, via the node tuning operator that might be affected by a REL upgrade or a REL version change, just be cognizant of that. Let's see, just scrolling through chat real quick. I'm going to trust you to raise up any chat. There was one that I almost missed. It was from KarateChop, plus one, I just wanted to say that username is from KarateChop. And it's, has anyone gotten OKD to work with the Assisted Installer? I don't think so. I would check on, so the Kubernetes Slack, so kubernetes.slack.com. So there is an OpenShift users room in there. I would go and check in there. I know there's some folks who have been trying it, but I haven't been following it closely, so they may have succeeded. Awesome. And I think we're caught up. I answered the, Rafael was asking about defining Ingresses. Yes, you can define an Ingress. Just know that, like, you can define an Ingress all day, but a REL is going to be automatically created. So just so you're aware of it. Yep. Okay, so let's see. Moving on, how to rotate the kube-admin password? kube-admin, kube-admin.  So if we switch over here, and I'm going to post this link into Twitch real quick. I was actually going to point to this blog too. Yeah. So Andy Block, who has been a guest here on the stream before he was here when we talked about the registry and all that other stuff internally, the internal OpenShift registry. So he posted a blog here about how to rotate the OpenShift kube-admin password. So generally, you know, we make this kind of blanket recommendation to consider removing the kube-admin user. That being said, I don't know how many folks actually do that. So this was super interesting to me. And when he highlighted that he had posted this, like you see it's July 15th, right? So last year, middle of last year, of hey, if you want to keep the kube-admin user, but you want to change the password, here's how to do that. And it walks through that, you know, pretty base or pretty easily. So what we see here is this OC abstract. So it'll get the currently encoded password. There's no way that I know of to decode that or reverse that into something usable. And then he has some go code here that walks through how to change it. And if you're like me, I have go installed anywhere or anything like that, but he has this helpful go playground thing. So literally all we have to do is click run here and it will generate, if I can scroll up, it'll generate a new password, right? So here's the password in plain text. Here's the base 64 encoded version of the hash. And then if we flip back over here, he has a very helpful patch commands, where you just inject that. And then after a few seconds, everything, you know, resets in the auth controller and you've got a new kube-admin password. So I thought that was super interesting. There's also a KCS that we can dig up that's on how to create new kube-admin certificates. So when you deploy OpenShift right in the auth directory, you have that kube config. So you can rotate those certificates. You can also create new ones. So I can create one for my user account, for example, if you wanted to do that. We'll dig that up when we have a moment.  Yeah, he's a machine. So the last thing I've got here, and then Johnny, you wanted to talk about podman is, so we, I got tired of, you know, having to go to YouTube and search and find all of the, all of the stuff for our streams. So I created a short link. So very simply, if you go to HTTPS red.ht slash AOA, I just posted that into Twitch. So if we go to red.ht slash AOA, it's a very simple redirect over to our playlist here on, on YouTube. So for anybody who's curious, if you want to go and see Johnny and I embarrassed ourselves on a weekly basis and see the history of all of that, you know, there you go. Documented history. It's amazing. Yeah. So, and then Johnny, podman 4.0. Yeah. So podman 4.0 just released, there's, you can read the full release statement there, but there's the big changes that I took away from that are, they did a rewrite on the network stack for performance. So there's a lot of like major changes that are coming out in podman to just make it more, more better, but really just to make it, you know, more performant on, on the desktops or the client side rather. And then within there's a lot of support for windows and the macOS. So if you kind of kept up with the last release, they had, they changed it over to like podman machine to spin up a machine on the macOS. So that way you can run podman. So there's better integration with that where now you can use like a Docker composed with it. So it opens up the host socket to podman. And then on the windows side, they're running it all in a WSL too. So a lot of big changes. If you're a windows user, now you can use podman pretty much exclusively. I think and then I think it's actually in there in the next release 4.1. There's going to be more support for volume mounts on windows machines. So, you know, stay tuned. It's coming. Podman is a great tool to use, especially if you're trying to get, I don't want to say it in a negative light, but if you're trying to get rid of Docker or you're trying to go to like a, you know, like a rootless runtime, then podman is definitely, you know, pretty good for that. I'll tell you I, so inside of my house, I use, you know, certain things for just convenience, right? So Christians helper node being one of those. And I use a lot of podman based services, but for some things I had been avoiding moving from Docker to podman because of Docker compose. I have them in, you know, a Docker compose format. Well, it turns out that as a podman three, like three or three dot one, there's like a podman Docker service and socket you can start. And then Docker compose just works with podman. I thought that was awesome. So I've been converting everything over to run off of podman inside of my, inside of my own lab. That's awesome. Yeah, I love podman. I've been using it for a long time and it's just, it blew my mind when I was like, what's a podman? No way a pod is like. So it's pretty awesome. Khalid, we do have something similar to that coming up. I don't know if it's going to be micro shift on podman, but it's definitely going to be micro shift. So we do have an episode coming up on micro shift. Maybe we can nudge them a little bit to try and do some podman magic. Yeah, that's a mea culpa on my part. And that's, I had reached out to the micro shift folks and said, awesome if you could join us. And they said, sure, we'd love to join you. March 2nd sounds great. And I completely forgot about it. We had scheduled and prepared this stream. And then yesterday they reached out and they're like, Hey, are we still joining? So we've, we've, we've rescheduled them. They'll be joining us in two weeks. So next week, I don't remember what next week's topic is, but we've got a bunch of interesting stuff coming up. We've got service mesh coming up. We've got micro shift coming up. We sat down and did our, our, you know, kind of loose penciled in planning for out through July, I think yesterday. So yeah, be sure to subscribe to whatever channel you happen to be following us on cause you'll get the full list of all the stuff that we've got coming up. Yup. And Pete real quick, I'm sorry. Pete, just so you know, I don't know if they renamed, I don't think they rebranded podman, or Docker compose to podman compose. The way that the, the really statement read is that essentially you can now run Docker compose against podman on a macOS and a windows machine. Yeah. And Docker compose like regular Linux Docker compose. It's just a, like you literally run Docker compose and then whatever magic in the background happens with podman and the podman Docker socket slash service. And it behaves exactly the same. So I, I'm not a super advanced Docker compose user, but hopefully it'll hopefully it will work for most of those, if not all of those use cases. Micro shift is open shift on arm. I don't know if it's exclusively arm or not Christian. I know most people think of it in terms of arm and things like, you know, a raspberry pie and stuff like that. It's a good question. We'll have to pencil that one down for when the micro shift guys come on. Yeah. It's definitely, it's definitely geared towards the edge target, right? That's, that's really ultimately what we're trying to get after with the micro shift is just getting at the edge. Yeah. Yeah. Christian is reassuring me that it is. So I believe you. I believe you. Christian's angry. Yeah. All right. So that's, that's all the top of mind topics we had today. So let's, let's talk about open shift 4.10. As I said, there's a, you know, I'm looking at our note stock with the outline that we create for each one of these shows. And like, there's like 15, 20 things here that, that we can talk about. So I'm going to try and cherry pick some things out that I think are important. And, you know, for anybody who's listening to us, please don't hesitate to ask questions or, you know, ask for clarification on anything that's interesting to you as well. And to be clear, we also have at least one, and I think more like two or three streams planned to focus specifically on some 4.10 features. So one of those is metal LB using BGP. So sometimes you'll see this called metal LB L3 node or BGP metal LB stuff like that. So it's a, BGP is a, it's, it's a, it's a big topic. Right. Just, just BGP, right? Just the border gateway protocol, you know, routing thing and then how that interacts with metal metal LB. We didn't feel like we could do it justice crammed into a show like this with a bunch of other things. So we're going to have a whole show dedicated just to that. So stay tuned for that one as well. That, that could be next week or it could be in three weeks. I don't know. So with that being said, let's talk about a few things. So let me pull up. I've got an OpenShift 4.10 cluster here. So the first thing I wanted to talk about is, you know, real quickly, if you want to get hands on with OpenShift 4.10 today, anybody can do that, right? So let me share my screen here. And if we come back over to our tab here with the, where's my screen at? There we go. So I'm just, I'm back on my tab from console.redhat.com. So if I go to clusters, again, if I click create cluster, kind of doesn't matter which one of these I select. So we'll go with vSphere. And then I want to install our provisioned infrastructure. And again, it doesn't matter which one of those I select, I can select user provision as well. Cause what we're looking for is this download pre-release builds option. And that will take us over to exactly what it says. That's pre-release builds, right? So here I can download installer. What I'm actually going to do here is I'm going to copy this link address. And if I open a new tab and paste it in, I'm going to remove the file name, because really what I want to show you is this is just on the OpenShift mirror, the standard OpenShift mirror. And it takes us out to this OCP dev preview slash pre-release folder. And in here you can download all of those. And you can see currently it's 4.10.0 RC7. So you can kind of bookmark these links. I have a bad habit of constantly browsing directly to mirror.openshift.com and downloading these things. And the official way of doing it would be as I showed, which is going through the console. But anyways, so you can download this. Once you have that, you install it, just like a regular OpenShift cluster, right? It'll come out and it'll look. So this one happens to be on Azure. You can see that we've got 4.10 RC7. So as a cluster, I deployed this morning. So one thing or probably the only thing that you'll notice about this, assuming everything works correctly, I had a failed pod down here for some reason, is if you go to cluster settings, you'll have an error here about how there's no stable 4.10 channel, which kind of makes sense because it's not GA. So yeah, but otherwise it should behave pretty much exactly the same as you would expect with any standard OpenShift deployment. Just remember being RC being pre-release, it's not a supported deployment. All right. So that's how to get 4.10 bits if you want to get hands-on. The next thing I want to show is I want to share my terminal window here. And the reason I want to share this terminal window is because I noticed something that was really cool. So if we look here and wait for that window to show up in the stream, did my console window or terminal window not show up?  Let's try and reshare it. Window, terminal window. And no, that's there. I don't know, Stephanie, are we... Do you want me to try and share mine? I have the install log if you want to see that. Yeah, that's all I was trying to share was the install log. Let me see if I can fail spectacularly. All right. There you go. Can you see it? I see your lovely face, but I don't see... Hey, there we go. Oh, that one's mine. Okay. Yeah, either one is fine. So let's see. So yeah, go ahead and scroll down. And so what you're looking at here, this is... I'm assuming this is the log? Yeah. This install log. Do you want me to go all the way to the bottom or... Yeah, I think so. I think what we want will be in there. This is the bottom of the pile. Yeah, so if you look there, there's a line maybe 10 up from the bottom, and it says, waiting up to 10 minutes, zero seconds. And one of the big changes here is you see it says until a specific time. I thought that was a really cool addition. Like of, you know, hey, it tells you before it just said, I'm waiting up to 10 minutes. And you had no idea when that 10 minute countdown started and when it's going to end. So now, of course, you don't have that issue. You can tell exactly when it'll expire and you can move on from there. The other thing I noticed at least with my cluster deployment is the the debug output was slightly less verbose. That could be just because my deployment went faster than normal. I don't know. But, you know, it used to be that there'd be like a huge slew of info mess or a debug messages as it's initializing the cluster. So I don't know if we changed the threshold for that or maybe just the deployment went faster than normal. But again, slightly different things in the deployment process. I mean, if you look here, the cluster deployed in 30 minutes, right? That seems at least a good four to five minutes faster than what I've seen in AWS consistently. Yeah, same for my, you know, mine deployed to Azure and it deployed in, let's see, 41 minutes. Yeah. And that's pretty fast for Azure. That's between 10 and 20 minutes faster than normal for Azure. I don't know why Azure is slower, but it is. So, yeah, hopefully that means good things and it's not just a fluke, anything like that. So getting the bits, installing them, you'll notice some differences. Johnny, if you're connected in your cluster, one thing I noticed, I think this started with 4.9. Like if you do an OC get CO, it now shows additional details about each one of the cluster operators. All right, let me do this. Yeah. So there's that message column way out on the right hand side. Yep. Oh, that's nice. Yeah. Yeah. So if you hadn't seen that as it's going through and doing updates and upgrades or anything like that, it'll tell you specific messages about the status of the operator.  And then you have to go and do an OC, you know, describe or an OC get dash O YAML on the operator to see the specific status message. Now it'll print them right here. I found that super useful because my cluster actually broke this morning or didn't fully deploy this morning, I should say. So it's pre-release. It's okay if it didn't fully deploy. Yeah, exactly. No, this is nice. I'm glad we're starting to do some of this stuff, you know, because, you know, I think getting some of this output, especially when you're doing a disconnected install, a lot of times we'll watch this, you know, and we're watching these deploy or these operators deploy just to get an idea of like where we're at in the deployment cycle. So I'm glad that we're actually going to start getting some of this. Yeah, it's quality of life things, right? So yeah, making our lives a little bit easier, you know, one step at a time. That's right. Baby steps. All right. So let's switch back over to my browser here. All right. So this is again, just my four dot 10 RC seven release. One of the things I wanted to highlight here is, and let me switch over to the documentation tab here. So CSI is now largely generally available inside of the project as a four dot 10. So if we go to understanding persistent storage up here and nope, that's not the one I want. I want this one. I guess this one. Yeah, if we scroll down through here, so we've got all of this CSI driver volume snapshots and all that other stuff, dynamic provisioning, you know, this, this talks about each one of the drivers and what they're supported with, right, which features are supported. So the slide has the specific drivers that went GA. I want to say all of the ones listed here are GA, but it will tell you at the top whether or not they are. So if I select Azure disk, for example, you see there's no big red box saying that this is tech preview. So we can see here it defaults to using an entry non CSI plugin to provision Azure disk storage. So if we look at my cluster and I come over here to storage classes, you'll notice that I now have four storage classes instead of just one like I used to before. So previously I would only have this managed premium storage class. This is the entry provisioner that's happening there. We now have these new CSI storage classes that are here. You can of course change this to the defaults, however be or change the default to be one of these others. You can also change different properties about them. I don't know what the properties are, but let's say you wanted to change the reclaim policy to retain or something like that or the SKU name to standard or whatever the other disk type is. So you can change those, but just like with all of the storage classes that are deployed and managed at install time, the cluster storage operator will ensure that they're always present. So if I were to go in, for example, and I'm going to put my money where my mouth is, if I were to go in and delete this guy, what we should see happen is it'll get recreated if it even deleted it at all. Are you going to delete for me? But the issue with that is when it recreates these, it will reset their values. So for example, if you change the defaults to say this managed CSI and then you go in and you delete this managed premium, when it recreates it, it's likely to set it back to the defaults. I say likely because that used to be the behavior. I don't want to assume that it's continuing to do that. But just be aware that it could cause an unexpected result for some of those. So CSI GA, I'm super happy about that. You know, as you saw over here, if we look, where did that go? If we look at this configuring CSI volumes, this adds a bunch of different capabilities, not the least of which snapshots cloning and resize, depending on which driver you happen to be using. So you see, I'm using Azure disk. We can do all three of those features, all three of those functions. So really, really useful. So the Rev, Red Hat virtualization, that CSI driver has been GA for a while. So that's not a new one. But I know the Azure disk one is new. AWS is new. Alley cloud is new. Open stack sender has been present or supported for a while, as has the OCS ODF one. So you'll notice that VMware vSphere, there's no checkboxes here because it doesn't support these features, but it is also a generally available CSI driver. And I want to take a minute to talk about that. So it's safe to say that, you know, there's a lot of open shift on vSphere deployments out there. And a lot of folks have decided to, and rightfully so, there's no issue with it, deploy the upstream VMware CSI driver into their cluster. It'll absolutely work. We've talked about it here on the show at least twice, if not three times. And so far as we know, it works great for folks. So as we move towards a future where entry drivers simply don't exist, that means that Red Hat will be deploying as you're now seeing, literally just with this Azure cluster, that we will begin deploying CSI drivers out of the box. And those CSI drivers are supported by Red Hat, which means that like we do with everything, we create a downstream of that. And the same is true for AWS and Azure and vSphere and all of those others, right? We have an open shift VMware CSI driver. So where this gets complex is in a couple of different ways. So first, I've mentioned before that we all need to take our clusters that have been created in 4.9 and earlier and begin doing VM hardware upgrades. So with VMware, the CSI driver requires that we use VM hardware version 15 or later, which also means you need to be on vSphere version 6.7 or later. So step one today, as you're making your upgrades into 4.10, as that CSI driver goes GA, we need to start doing that virtual hardware upgrade process. And that'll be fully documented as when it comes out. So with 4.11, what you're going to see is a migration process become available. So it's in tech preview today and 4.11, it's expected to go GA. So what that means is today you're going to have some volumes, you know, if you're upgrading a cluster, you're probably going to have some existing volumes that are either provisioned by the entry provisioner or by the upstream VMware CSI provisioner. When you upgrade to 4.10, no change, everything works great. That's phenomenal. You will have the option of deploying the Red Hat VMware CSI provisioner, the OpenShift VMware CSI provisioner. When you do that, it will uninstall the upstream VMware one. In 4.11, we will GA the ability to migrate entry volumes to CSI volumes. And that's not just for VMware, that's for a lot of the other provisioners as well. But just be aware that upgrades and the console will tell you about it, right? So if you saw over here, the console will tell you if it detects something, and in my instance, I've turned on the tech preview thing, but it will tell you that hey, you cannot upgrade and here's why. So just be aware that like to go from 4.10 to 4.11, when the CSI provisioner becomes mandatory, you'll have to have all of those prerequisites met. So VM hardware version 15 and all that other stuff. So again, please prepare now, be aware of this. Hopefully it is not disruptive. You can always ask questions, open support cases, all that other stuff. The storage engineering team has done a great job of preparing us for that. So and then 4.12, 4.13, it's the you know, it's the VMware, the OpenShift VMware CSI driver, all the time. So looking forward to that should be interesting. And CSI is, I would say, you know, maybe I'm slightly biased because I came from a storage company, CSI. The CSI provisioners are way more powerful and way more capable. Yeah. All right. I've spent long enough on that topic. Hopefully if anybody has any questions or anything like that, don't hesitate to reach out. I have this conversation semi- regularly, so I feel comfortable having it again and again and again as the case may be. All right. So what are some other things that are changing or that have changed with OpenShift 4.10? So one thing to note is that NM State is now GA within Asterix. So what do I mean by that? So the Network Management State Operator, NM State Operator is or has been GA since OpenShift Virtualization was released in 4.6, but only when used with OpenShift Virtualization. So it had to be a bare metal cluster or a physical cluster. You had to deploy the OpenShift Virtualization Operator. It would deploy NM State as a dependency of that. With 4.10, they're broadening that to include all bare metal clusters. So you can deploy onto bare metal. I don't know if I have it in the docs here or not. Unfortunately, I can't search these docs because they're the pre-release docs and the search doesn't work because it's not public. Let's see. Networking. And I want Kubernetes. Nope. Kubernetes NM State. Yeah. So you can see here in production environments on bare metal platform installations only. For other platforms, it is a technology preview or continues to be a technology preview. So this is important and it's a big change for a couple of reasons. So first and foremost, you can, when deploying bare metal, you can actually introduce those configurations as a part of the install config.yaml. So this is kind of cool in that if I am deploying to a physical server that has physical servers tend to have 4.6, 8.10, 12 network adapters. And I need to create a bond and I need to create a VLAN interface and on that VLAN interface is where my cluster's primary interface is going to reside. We can now define all of that in the install config and NM State will apply that during install. So it makes it much easier than before where the only answer before was deploy the live ISO and boot to that and then use NMCI or NM2E to do that. So now we can do it kind of natively directly in the install config. I haven't looked. I have no idea if it's actually let's see if it's in the bare metal docs here. Installing on bare metal user-prision cluster on bare metal let's see NM State, no, it doesn't show anything. So simple config files. Maybe they haven't updated this docs page yet. So a little secret about the docs so if you're sharp-eyed I know it's hard to see on the stream I'm pointed at the 4.10 link. So if I were to just browse to docs.openshift.com and I select version there is no 4.10 So version 4.10 is actually protected by a ht password so not everybody can use it. But if you go to GitHub and the open shift org and then we search for open shift docs you can cheat. You can look at the pull requests that are in here and you can find pull requests for things that you're interested in, right? So let's see here, removing 3 network adapters from SRIOV. And inside of each one of these there's going to be a preview link. So I can click that and it'll take me over and show me the preview of what those documents look like. So I bring this up because I was actually using it to cheat for IBM cloud IPI so this was how I figured out how to deploy on the IBM cloud using IPI. So if you're interested, if you're one of those folks who wants to deploy a version of OpenShift pre-GA and you can't find the docs, this is one way to access those docs. I don't know what's taking. Netlify Netlify, whatever the services in Netlify I don't know why it's taking so long for that page to load but that is the way that we can see pre-release docs. So I won't... Tricks of the trade. Yeah, I won't let that sit there in turn any longer. Let's see, platforms what do we want to talk about next, Johnny? Oh, here's one that I think a lot of people will like. So I'm going to share my console again and switch to here and by the way, it's okay that I shared my Kube admin password here because I used dandy blocks process to change it before I knew I was going to be showing it. That's awesome. Alright, so inside of here I've got a I'm just going to do a simple let's see OpenShift install creation so I'm using the 4.10 RC7 I'm going to do an OpenShift install create Ignition I want install config so I'm just going to generate an install config file so yes, we're going to use that I want to deploy to vSphere my vCenter is going to be this guy and what I'm choosing, I'm not actually going to deploy this so what I choose here is really irrelevant get there eventually so pull secret another little secret if you don't want to use a real pull secret is to grab so the okd docs it fits the right JSON format to pass whatever check they're doing here without actually using a pull secret so Johnny for those of us who share these things on broadcasts all the time so what I've got here is I now have an install config.yaml if I do a less on that it's pretty standard what's inside of here so let's exit out of that and what I want to show you is if I do an OpenShift install explain install config platform.vSphere and so if you're not familiar with the explain so both OC OpenShift install all of the commands have that and it does exactly what it says it explains as you walk through the yaml tree of whatever object you're looking at it tells you what each option each value happens to be and what I want to show you here is this disk type option so historically if you deploy a OpenShift in particular IPI cluster it will create that it will deploy the OVF and it will deploy it to a data store that is going to inherit the disk type as we're calling it here but really it's the yeah we'll call it disk type from the data store so if it's a block based data store it will be a quote-unquote thick right if it is a file based data store so NFS it will be a thin disk so this is sometimes good and sometimes bad if I'm deploying to an iSCSI or fiber channel data store that's 2, 5, 10 terabytes in size and I'm wanting to provision 501 terabyte drives from my OpenShift nodes that's a lot of space to consume initially without it may or may not be what I want maybe I want thin provisioning so there's ways that you can work around that by converting the disk type and all that but they added in an option so disk type is thin so what I want to do here now is I'm going to edit or install config all the way down here we'll add the option of then now when I actually go to deploy that what I should see here in just a moment if this works there we go so it's going to download that dva file real quick hopefully real quick hopefully it's not affecting my bandwidth for the live stream that should be fine it's gigabit internet it's fine take it down the internet right now that's what's going on so Khalid how can we automate the UPI deployments IPI? no I'm kidding so there are a number of folks who have published unofficial methods of updating UPI using things like Ansible and some other ways as well so one of those I think there is one let me check yeah I used terraform too so one of those I think yeah let me switch which screen I'm sharing here or which window I'm sharing I want to go with this guy alright so all I did here was I went to github.com slash redhat official which is the official github account for redhat as it says here but many of these repositories are created, managed, and maintained by the community of practice which means that they are redhat people but they're creating unofficial and unsupported type of offerings so I bring this up and they should have a disclaimer they usually always have a yeah so it's most ideal for a homelab and proof of concept scenarios so here's one example of vSphere UPI automation I think this one is Ansible based if I remember correctly I'll post that link here into Twitch there's a number of folks who have created these types of things look in the community you could probably do a Google or DuckDuckGo search and find a number of those which tends to be one that I remember regularly so what I want to check here is what's going on with that deployment let's do a refresh yeah so here I've got my virtual machines from my test cluster and if I go to edit settings here and I click on hard disk you see that it is a thin provision disk yay and normally I would not be able to deploy a cluster into that that data store because it's a nice SCSI data store and it's just not big enough to hold all the disk for all the VMs so I'm super happy about this I know it's something that we've been asking for for like four releases or something like I think since IPI came out so good news it's there you can now thin provision your open shifts deployments I would feel slightly guilty if I don't remind folks however that thin provisioning is a concentrator of IOPS so remember that your data store the underlying data store does need to have the capacity not gigabytes capacity but IOPS and latency capacity for that workload etcd is notorious for loving wanting low latency IO if you're now cramming because thin provisioning you're cramming 2, 3, 5, 10 shift clusters and they're at CD into one data store just be aware of that impact to performance so I'm happy about this, love it for my lab but I know there's a lot of folks who also use it and wanted it for their enterprise environment too Walid favorite feature out of 4.10 Johnny I'm going to throw you under the bus first so I've got two one because I'm biased and the other one well really both because I'm biased but the first one is like the mirror registry coming out and being a part of that to help the disconnected deployment trip because it's near and dear to my heart like it's something that I've struggled with for a long time and I know people struggle with it all the time and so I think just that having that component now that's a legit registry that we can put out there and actually do stuff with I think that that's an awesome update and then it's a tech preview feature that's coming out and that's going to be OC mirror and I'm biased because I know all those people that worked on that project and they are doing some awesome work they're doing God's work you know with the OC mirror and getting all the images and stuff like that collected so really happy that they were able to get that going because it's an awesome feature Yeah I'm going to go with two as well so one is Metal LB using BGP that one's really cool and it fills a huge on-prem gap you know hey I need to if anybody watched the level up hour earlier today they were showing you know SQL server and you know like hey I need to expose ports 1, 4, 3, 3 you know the standard SQL server port well you can't with a route because that's limited to port 80 and 4, 4, 3 you know doing it with a node port you know where it sucks that's just it's difficult it works but it's not the best way so Metal LB and especially with BGP mode now I can actually expose all of the ports that I need and it's a true load balancer type of thing Christian throughout the phrase earlier of it's a true load balancing type of capability that operates at the network level I will say Andrew, old school Andrew having worked with a lot of smart network guys but that being close to a decade ago I had some trepidation about BGP and you know what that looks like we've all heard stories of Facebook and one of the CDNs like multiple really big, really smart sets of organizations have broken things via BGP so I will say I absolutely have some trepidation about that but the capability there is just so powerful yeah so the other one which is one that I'm working on or was working towards is EOS to EOS updates so 4.10 is now the first EOS after 4.8 so remember even versions are not EOS so if you're on 4.8 you can do an accelerated update between those two versions so we've talked about that before the short version is 4.8 to 4.10 upgrades is it pauses updates for compute nodes and it does for the control plane it does the 4.8 to 4.9 to 4.10 update and then for the compute nodes when it's for lack of a better term unpauses those it will do a single reboot to go from 4.8 to 4.10 so particularly for folks that have larger clusters and by larger I mean 10, 20, 30, 50 nodes rebooting all those nodes can take a long time even more so if you're running hardware even with just a handful of physical nodes in your cluster hardware takes ages to reboot and physical servers with a terabyte or two or six of physical RAM that's a long time to reboot so I'm super happy to see this it'll hopefully bring down those update times for folks so yeah that would be my too yeah that's the the U.S. updates and then the other one is the conditional updating I think that that's an awesome future as well like to finally give people that warning like hey this could go really bad you could do this but that's a good point so that came out of when we were having we the royal of both Red Hat and VMware we had some releases where specifically with VMware the rel kernel driver for vmx net3 was just flaky and you remember like that was what caused the I think it was 4.7 or 4.8 release into stable to take an extra two or three weeks because we had to sort out that problem and get it resolved before we could turn on stable updates so now with conditional updates basically it'll say before it was an all or nothing even if you're deployed on to OpenStack or Rev or AWS stable was blocked for everybody because of an issue on one platform now it's hey there's these known issues are you sure you want to update and yes I want to update that doesn't affect me type of thing I think you could always technically do that it was just a much more obscure way of doing it it wouldn't give you those warnings it wouldn't give you that there's these other things that are affecting other things so or these issues that are affecting other platforms so yeah conditional updates was another one see I'm looking through a list when you see me look off to my left that's what I'm looking at is our list of things to talk about today so Johnny other things for you I'm going to quickly look through chat um yeah I think that the I mean just really kind of like we can just touch on is like the start manager support where you know it looks like they're the quote from the document was like you know they're making certificate management of first-class citizen and open shift which I think is like long overdue you know I think that we've always handled the API and internal certificates really really well and I think with open shift for especially the way that we've simplified updating the ingress controller and then updating the API search is really great but having something that can like automatically renew those for us you know like like start bot manager you know I think that that's going to be really great for um you know all of our users and administrators out there that hate certificates like you know 90% of us so I think that that's an awesome future that's coming up and it's tech review right now and so I'm just looking forward to seeing that mature and really get rolled out and it's going to be a nice future the open SSL command is like that's arcane magic to me I think oh yeah I've been using that command for 20 years and I still just barely understand it yeah it's I know like a handful of them I can do like the s-client connect and then I can do like if I need to look at a cert to see if it's valid or not I can do that that's that's about the extent of it yeah no and the the ice guzzy cli is about the same right oh my god don't even give me a start on that yeah uh let's see uh Hamantha an apologies if I am butchering anybody's name um latest open shift versions are draining resources in the viewware environment is there any plan to optimize so I will say yes but I don't know what I can share on that respect so I would suggest looking at the most recent what's next post so let me find that again and post that link and do so let me post this into which real quick so I would encourage you to look at the the what's new or excuse me what's next the roadmap presentation there should be another one coming up relatively soon I want to say in the next maybe six to eight weeks and what you're going to look for is things like composable open shift and all of that so yes it's something that we are interested in that we're looking at there's also efforts we talked about micro shift before um Christian highlighted its arm only though but um you know we are aware and they are working towards reducing that footprint and making it easier to reduce that footprint so you know hey I don't need this service let's not deploy it type of thing um let's see we got a question uh it was from um at least that's Sean I'm gonna go with Sean uh and it says which search do you recommend for a beginner to get into open shift I can jump on this if you want or if you want to give your insight that's cool too um I I agree like it with uh uh rock count you know CK is a great place to get started especially if you just want to learn kubernetes you know as as just a platform um CK and CK ad are both really good um CK ad is more geared towards you know the application developer so you're gonna spin up a lot of config maps a lot of secrets you know attaching them to um you know your applications and stuff like that uh CK is more administrator focused so you think about like updating your cluster adding nodes to a cluster deploying clusters using kube adm where if you look at it more red hat specific that EX 180 is very podman and like early open shift focus and then the I think it's 280 is uh where it's just it's the open shift admin but it it kind of jumps like into some like you know almost like you have to have some existing knowledge to get going um but it does give you like an excellent rundown of a lot of the features within open shift and you know like routes and the things that are specific to the open shift API um and then for an application developer in open shift there it's the DO 288 and that's the that's a CK ad equivalent and and open shift so I think any of those are a good place to start um and they're all great courses you're muted Andrew. Thank you I hear myself perfectly fine it was a phenomenal conversation yeah I'm gonna share my browser window real quick um so let's see and we want to go to learn. redhat.com um and the reason why I want to go here this is kind of my default for all things um so if you look at the uh yeah it's been so long since I was here that I don't even remember now um we want to find the open shift certification path um might be better off just searching for it but um skill paths here we go so this kind of walks you through all of the things that Johnny just said at a minimum I always recommend folks do this DO 080 you see it's it's free anybody can go and take it you don't have to pay anything it kind of gives you that just getting your feet wet getting you know started with open shift and containers and Kubernetes and all that other stuff and then you can kind of go from there um but you see the the X 180 which I think while he'd recommended I think for a lot of people unless you're going to be it depends on your role right if you're the lead administrator for an open shift cluster or clusters then yeah you're probably going to want to go with something you know pursue all the way up to like a 280 or something if you're a junior administrator you know maybe start with the EX 180 and you know learn from there um the good news is and as Johnny said um and I think some others have said as well open shift is Kubernetes which means if you go and you do like the CKA the certified Kubernetes administrator exam like 98% of that is going to apply to open shift as well um really it's it's only the things like routes and stuff that we add on top that yeah it translates extremely well and and vice versa right if if you go and take you know the DO 180 or you know the other courses here um most of that knowledge is going to translate if you want to go and also get a CKA or a CKD or something like that so one thing that Andrew did earlier that I think it's a really great it's built in and it's been there for a long time is the explain option right so like kubectl explain if you do that with the resource type so kubectl explain deployment and then you do dash dash recursive that will actually give you everything that like what he was able to dump out from the install config so when you're going through and you're like oh man I don't know if this is supposed to be a map or if this is supposed to be an like an array or you know like what what's the actual field name that will give you all of that information and then that works with OC that works with kubectl which you know Andrew showed earlier I think it's a it's a great feature that a lot of people don't know about yeah and just to um because I learned this remember when we had Ali on you can also go in into here and you can you can create those inside of here and it will pre-populate things for you so if I create a builds right it's got all of the schema it's got everything that I need over here I can click details and it will tell me exactly what I need in here and you can even create custom examples and stuff like that if you if you need to so I if you want to use the GUI to do it great if you want to use the CLI to do it great but that information is available in both places somebody who was it somebody was just saying that they miss the rockhound I missed the instant learn.openshift instances so they should still work so if anybody who's curious you can go to openshift.com if I can type it redirects now over to developers at redhead.com and if you scroll down we have this interactive lessons I can hit this openshift 4.9 playground and then I hit this launch button and I always get confused by this because you see that that's not visible here there's nothing that says oh scroll down a little bit and click that button so sometimes you'll get you'll be able to start one of these right away sometimes it has to spin one up in the background but usually it's not more than a few minutes and then you can click in and do what you need to do I use those all the time if I don't have a lab deployed or a cluster deployed in my lab or on AWS or something I go in and click these and I use them to answer questions and figure stuff out so yeah I'm always happy to evangelize for the stuff that they've got going on here and if we go back to the all courses you see that they've got just a massive number of things inside of here here poke around the openchef web commands web console there's the playground so anyways let's see in the interest of time because we're six minutes after 12 here on the east coast it's one other thing that is exciting to you Johnny let's see oh the ACM hub and backup that's going to be awesome so we had oh man I can't remember her name no dang it it was just at the tip of my tongue we had her on a couple weeks ago I was thinking Antoinette okay good so we had a net on a couple weeks ago and you know she went over the backup and restore using ODF and the cluster you know I think having you know multi cluster support would use an OADP I think that's just going to be a huge feature for our customers especially as we start deploying you know in number of clusters going down the road we don't have to worry about them blowing up you know I mean we can actually recover from something like that yep so I'll I'm going to cheat I'm going to say two things one is GA and one is tech preview so GA is GA is ARM yes so OpenShift on ARM in AWS as well as you know bare metal ARM is GA I think that's awesome I agree I have a good friend Derek and he's all about ARM deployments and so I think that you know he's probably getting excited yeah it's I know for AWS it's one of those like you know hey it's dramatically cheaper especially for the type of stuff that I do right I'm constantly creating and destroying clusters so you know we can we can spin those up for much less so the other thing so tech preview and the reason why if you saw earlier that I had the tech preview feature gate enabled I didn't have time to deploy it because it's actually dependent on and I didn't realize this at first is dependent on ACM 2.5 but they kind of talked about it during the what's new presentation but I want to re-highlight it and that's like if you've ever used cockpit and you've connected cockpit to multiple servers right up in the upper corner you'll have this drop down where you can say which server you want to connect to and you know I want to connect over to you know server X and it's all the cockpit options but you're just using one cluster as sort of the hub for that so you'll be out you can do the same thing with ACM and ACM 2.5 and OpenShift 4.10 in tech preview so effectively if you have more than one cluster to find an ACM you'll have a little drop down up here for the console and you can say which cluster you want to connect to so it's been a while it was like one of the first shows you were joined me as the co-host Johnny where somebody was asking you know how do you keep all of your cluster console straight and we were you know showing using banners and stuff like that well now you can see like there'll be a drop down up here that'll tell you specifically like here's the cluster you're on you'll have access to all the normal stuff inside there I thought that was super cool it is really awesome DMI 3 can we mix architectures control plan on x86 and worker on ARM not yet so that is not something today they do need to be homogenous but it is something that they're looking at I think ARM is the first one that they're targeting rather than P and Z and that is in the roadmap if you look at the roadmap slides so I mean I don't know what I don't know when they're targeting it but it is in there will image content source policy support name tags I think you asked that at the beginning rockhound and we might have missed it so thank you for asking again so I asked this the other day and I believe the response was actually no we asked it when we were talking about or when we were talking to Daniel and Daniel said maybe from what I remember I'll have to double check with him and follow up so I don't want to say yes or no because I don't remember the answer that he had and I don't want to give you an incorrect answer so please please watch for the blog post or the next week we'll talk about it in the top of mind topics just to confirm our hope nine yeah thank you for linking that deck or that slide in the what's new section so that is the high level overview there let me copy that link address and paste it in here yeah if you look at that so here's that upper right like I can maximize this or full screen it no so here's that upper right hand corner just like with cockpit and you can see that you can see that you're connected to the cluster you're connected to and have access to all of those features all the things that you're used to managing so I was hoping to have it deployed and ready today for this for the stream and I didn't realize that I had to deploy a pre-release version of ACM in order to get it 2.5 so I just ran out of time so hey Zandersen and game development so we will take that into into account so with that being said I don't have anything else so we'll post a list of the list of things that we talked about into the blog post we'll link into where in the stream we talked about all those things there'll also be a few other things that we didn't cover here as well as we've got several other again future streams planned around specific features and expanding on those in more depth like Metal LB and BGP so in the future or excuse me for now if you have any questions please don't hesitate to ask those here in chat get those last questions out we'll do our best to answer those we have two more so in the meantime if you don't want to ask publicly here in chat if you don't think of something as you're watching this afterwards maybe not live on one of the recordings please don't hesitate to reach out to us via social media so you can reach me at Practical Andrew just like you've seen my username on Twitch there on Twitter you can also, Johnny and I have started stalking Reddit so the OpenShift subreds and of course on email so you can reach me andrew.sullivan.redhat.com so you said there was two questions Johnny there are, there's one from Tiger about the Kubatman password rotation I tried to post a link in YouTube earlier and it's not getting redirected so I'll repost the link and then the second question was adding the windows nodes to existing clusters so I don't know if Christian is still on I think I mean as far as I know that bring your own windows node is GA I think Christian would be the one to confirm that because he's the he's the windows nodes guru I like to throw him under the bus and then in the RockHound 1 just asked if the offline installer IPI for OCP is coming as a single package that's a good question I don't know we, you and I I think briefly talked about something like that yesterday Johnny just kind of unofficial of I suspect so to be clear this is Andrew like assuming guessing rates not authoritative in any way I don't know if we will consolidate more than the OC mirror set of commands so basically what we're looking at in 4.10 in a mixture of GA and tech preview is there is the open shift mirror registry so one command to go and deploy a standalone query instance then there will be the OC mirror command so now it's mirror all of the data that you need into that mirror registry pick that registry up and move it to the disconnected environment and now you have everything that you need over there I would love for all of that to be in one Andrew's personal thoughts and Johnny I mentioned this when we talked yesterday about it is open shifts is big a full open shift if you were to download all of the images it's something like 10 or 12 gigabytes so shipping that in binary or one tar G zipped package that you put on to a host would be a lot and you know it's one of those the packages the images are constantly being updated so that package would either always be out of date or you're constantly having to re-download a 10 gigabyte file that has everything inside of it so to me that makes it impractical as much as I would love to see something like that yeah I think with OC mirror they're doing some work which actually helps with the compression and the way that they're you know like if an image has UBI8 it'll only pull that once and then essentially cache it for every other image so it does help shrink it down but to your point from yesterday to today the release candidates for open shift they changed overnight so it's like every day you'll be deprecated essentially or out of date yeah rock on when you're air gapped this is a challenge in general very much so I know that pain so and to that point too right like if you're pulling down a 10 gig file you're going to have to split that file you know most likely to get it through your security so that way if you have to burn it to DVD or if you're that restrictive which a lot of our customers are you know I mean that's a lot of overhead on the administrator side as well that you know I think can cause some problems it does cause problems Fahad for the mere registry would you need a dedicated server no I think the minimum requirements are 2 CPUs and 8 gigabytes of RAM for quay excuse me but other than that it just it literally spins up quay running in podman so you can do other things with that server if you like particularly if you have more than the bare minimum requirement or resources okay well thank you everybody again if you have any questions anything that we weren't able to get to today please don't hesitate to reach out through any of the very various platforms email is always great andrew.sullivan at redhat.com or johnnyjonny at redhat.com happy to answer those questions happy to do what we can to help you out today's show I hope has been helpful if there is anything that you are curious about around 4.10 any features functions capabilities etc that you're interested in send them over to us like I said we've got we plan on spending some time on 4.10 and 4.11 when it eventually comes out so if there's things that are interesting to you by all means don't hesitate to reach out that being said be sure to subscribe for our shows in the future on things like micro shift service mesh etc I'm excited about those shows service mesh is something we've been talking about for like a year so and micro shift is something that's just exciting in general so mm-hmm all of that being said thank you Stephanie in the background appreciate your help today and Johnny I'll leave you with the last word yeah it's great episode today you know everybody thank you for participating all the questions are awesome it makes these things fly by and you know we love obviously talking about the tech stuff so you know glad glad that you guys are showing up and we're able to help out so see you next week


SEGA DRUNK The Super Nintendo got tons of Capcom beat-em-up arcade ports through the early 90s, everything from Final Fight to Knights of the Round to King of Dragons, but there's one that only came out for the Sega Genesis, the Punisher. My only guess as to why this was only ported to the Genesis is probably due to the quote-unquote mature content here, and the Genesis was perceived as being the more adult console, so to speak. After all, they got the blood in Mortal Kombat before the SNES did. But yeah, oddly enough, this wasn't even ported by Capcom, but by Sculptured Software. I had no freaking clue this game even existed, thanks to Anthony from my Patreon page for requesting it. This is a fantastic old-school Capcom style beat-em-up with all the usual bells and whistles, tons of weapons to use, everything from knives to M16s to lead pipes to grenades to a fricking battle axe, and you can actually use them on bosses too, which is nice. Sometimes these games frown on using special weapons on bosses for whatever reason. In addition to that, all the classic beat-em-up maneuvers you're used to, if you've played, you know, practically any 16-bit beat-em-up that's ever existed, they're all here. You can grab these guys and spike them into the ground, do rolling kicks, a leg sweep attack. I mean, not that you'll need to do any of these moves because there's so many weapons in this game, but yeah, despite this being ported by Sculptured Software, this plays exactly like how you'd expect from a Capcom beat-em-up. Of course, if there's any comic book universe that lends itself beautifully to a beat-em-up, it's the Punisher. For example, there's a section in every level where you end up with a pistol out of nowhere for some reason, and you just shoot as much as you can until you run out of ammo. Why? Who cares? It's the Punisher. There's no rhyme or reason to it, but of course he just has weapons materialized from nothing so you can shoot people. But yeah, he's not the only character you can pick from. You can also play as Nick Fury. So yes, this game is 2-player co-op, although if you want to play as Nick Fury in single-player mode, you have to set it up in the options menu, which is weird. Both guys have a wide array of moves and really aren't too different from each other. Same size and range of motion and all that, but I guess it's nice to have the variety for aesthetic reasons if nothing else.  I don't know if he's just really into his job or if he just doesn't care if he's zonked out on opioids or something like that. It's great. I love that guy. So yeah, the Punisher for Sega Genesis certainly isn't anything new or revolutionary and make no mistake, this definitely is not as good as the original arcade version. But it's a perfectly fun multiplayer beat-em-up with a lot of weapons and carnage. Even the sound design here, which can be iffy on some Genesis games, is really well done, especially the gunfire sound. So yeah, if you've grown a little tired of stuff like the Streets of Rage series or if you're just a fan of comic book games or Capcom-style 16-bit beat-em-ups, go check out the Punisher.