George Bruna with a 21 report. We're at the 21 convention the final event of the decade, and I'm talking with Ken Curry Welcome. Thank you George. Appreciate it. Yeah Licensed marriage and family therapist. Yep married 35 years and three kids. Yeah, you have something to say about marriage, don't you? Yeah, it's a it's a pretty awesome thing. Yeah, it has changed my life It's been a source of incredible significance to me.  Okay. Yeah, excellent. That's how mine are my oldest is a daughter and then two sons. Yeah Yeah, excellent. So you did a workshop at the Patriarch Convention. I spoke at this one You are a rising voice and your voice is important in this organization Tell us about what angle you're coming from your work How watchers will benefit from this since digital is forever and people are going to be watching this five and ten years from now Yeah, so I feel like I have a significant thing to speak on that has to do with the substance That's underneath the surface So much of what is said in at the convention even even the at the convention I feel like It's it's good. It's clear You know how to how to relate to women how to talk to women how to dress how to there's so many things that are they're Significant, but I feel like the thing that I want to bring to this whole thing is Something under the surface Something that has to do with the my core as a man The depth of who I am as a masculine individual So so like my talk this morning Really was about my identity and about how our identity is developed and how we listen to so many stories and narratives about who we are growing up and there's so many voices out there telling us who we are and what life's all about and Really powerful part of moving into becoming a strong Man with a really solid core of strong identity is being able to I call it That was in the title of my my talk was Who's telling your story and the whole concept of I? Need to be the one telling my story. I need to be the one to say this is who I am This is my identity and so I think at the core of everything if a man knows who he is In his being that he's good and that he he desires great things for himself and for his family for the world I just think it's a really really powerful thing for a man to start with that And starting to develop that really strong sense of self if a man doesn't tell his story By design then by default who ends up telling that story by default then Kind of part of my talk was it starts with your family of origin Your family of origin kind of teaches you the rules and the role that you play and and you're part of a system of Of that family and you pretty much have to go by what they what they say what they want and then different voices like school and teachers and Our culture and church and so many voices out there telling you what's good and what's not good Our culture is really horrendous right now what it's telling us about men men are toxic men are abusers men are oppressors and And it's just like is that true? And if I actually just believe it and kind of just go without being awake or aware of what's being told to me Then I end up with this identity And I'm kind of a I'm not good enough Something's deeply wrong with me or something's flawed And that's what so many men kind of have that general sense because they've listened to those voices so much Can you imagine if it was the other way around and there was a movement in the media talking about toxic femininity? Would that ever happen? Well, it kind of seems like it's not going to happen because it's kind of in our That's kind of where our culture is right now I don't want that to happen. I don't think femininity is toxic just like I don't think masculine at these talks, right? But it's unfortunate that that's the that's the message that's going out to our young men. Yeah That's a shame Prior to our talk here. You mentioned order disorder and reorder Is that applicable in a man's life? Oh my gosh Talk about that. Yeah, so they so the primary order And that's kind of what I was talking about was these are the things that I believe when I grow up when I have a certain order Because we all grow up in a certain family system, you know, it's and it's ordered. It's going a certain way There's a point in time every one of us, you know, you go over to Billy's house You know as a kid and you're going man, his family is totally different than mine and and you start to realize There's something different going on over there What's going on with my family? And so then as you as you continue to age you start going what this order is Isn't necessarily as healthy as what I want. This isn't necessarily where I want to go or this isn't the life that I want to live and so then and Sometimes it's pretty tough The whole for this order piece because because something has to change and so the disruption of how this happens I work with a lot of guys. I do a lot of men's groups and And so a lot of guys will they'll go through the Change changing their heart changing their soul changing where they want to go changing their narrative, that's probably the most important part of this and And and they start changing how they move in life. Sometimes it blows everything up Unfortunately, sometimes their wives go along right with them like mine did my wife totally went through the changes And we are loving it She is loving being married to me and we have a great relationship And so the thing about it is is that disorder Can be really rough, but sometimes we can actually go together and create a new order on a new foundation And a lot of a lot of couples that I'll work with, you know, they'll say we want back what we used to have and I go No, you don't you don't want what you used to have You want to create something new on a completely different foundation and that's really what we want to be able to do Interesting we had talked about Louis Smedes who said my wife has been married to five different men and All five of those men were me were me There is no doubt part of the part of my talk this morning I shared a couple stories one of the stories was and it was funny because it really dated how old I am But the whole idea of going into a video store. Yeah, right? I Used to be scared So badly with what the girl who was sitting there what she would think of what I picked for a video Right, right, because I was so externally referenced and so worried about what other people think Yeah, and so that that that young man that boy in a sense I mean, that's who my wife married initially and who I am now Oh, obviously the fifth sixth iteration of me. Yeah. Yeah, she's walked through this whole thing Yeah, with me. Absolutely. That's I love that quote. Yeah Yeah, that it made an impact on me. Yeah, because usually There's a lot of people talking about the changes that women go through and even when a woman Goes through that period later in life. We call a change of life. You don't hear that much anymore people don't say that phrase What do they call it now menopause they call it menopause in the old days Remember they would say it's when I go through my change that's right. Yeah, and everyone knew what that meant and Everyone changes in a marriage. We don't remain the same It is that I actually think that's one of the most profound parts of marriage or purposes. Yes The whole idea of the design of marriage is to grow us up to maturity Mm-hmm, and I think that's the whole process that we're talking about Yeah, is that I'm going to become a much more mature individual Because I've been married and when you're both on board with that where she's growing. I'm growing. It's it's pretty amazing. Yeah It is a badge of honor when you said you've been married 35 years. I look at that and I think I Probably will never be married for 35 years. You know if I got married today I'm not going to be married for 35 years to anyone. You know just with My longevity probably not most likely unless I live to a hundred. I don't know happened to my grandmother She was married for 50 years or so to three different men. Is that right? Okay? And she went she lived to a hundred did she really she did so he already spot on. Yeah, right, right What is it like What can you compare a 35 year marriage to is it seasons? Is it stages? What would you What other picture can you compare it to? That is a great question. I don't that's a tough question to ask on the spot Yeah, because definitely a seasons definitely stages because I remember, you know The kids were little or even the season that we had before we had kids Which wasn't too long. It's only a few years and then having little kids having The kids in grade school and then we did in the middle of that we did eight years of foster care as well And so five eight years of that so that was a totally different season and then as the kids were Going through college Because we had like the double boomerang where they would keep coming back Yeah, right and now they're all totally out so But yeah, we're at a totally different season of emptiness now But it's I don't know what metaphor I would use but it's just like and I don't even know what's next because there's gonna be another season and Well, you're beyond emptiness now. Yeah. Yeah, and now we're waiting for grandchildren Yeah, and so that's probably the next stage probably is absolutely, but it's been a lot of fun You know, I've seen my role as you know helping my kids get older I think one of the key things was making sure everybody had a college education and So my youngest had December of 2015. I think Graduated and then my wife graduated May of 2016 and so it's like my work is done And so it was great. It was that was a transition now everything's gravy, you know Now, let's see where we go from here get to kind of coast a little bit a little bit But the cool thing was that now I can actually start to move forward With what my passion is. Yeah, you know, I feel like that was a great season of my life to Get all get my kids grown and educated and now I'm really moving into the whole idea of working with men writing Doing things like this and and it's just been wonderful to be able to given be given a really good platform to be able to speak To be able to kind of have a really strong influence with the men because I'm so passionate about empowering men It's just that is so important to me So it's just really fun kind of being able to have this opportunity. It's awesome Who was your target market when it comes to the male demographic the young man? College age the newly married man the young father Who who do you think your message resonates with the most? That's a great question because I wanted to just answer the answer is yes Men I just want to I want to make sure men are able to understand because the whole idea of my speech is I didn't really think of that it would be this group.  It was all young guys in the audience now. There is gray hair. Yeah in the audience a few guys Yeah, absolutely and so your message resonates with men in general And it's kind of interesting how a conference like this can touch a 60 year old man and touch a 20 year old absolutely Yeah, I think that's the brilliance of having the especially all the we call it the diversity of who we have because it's like Man, everything has diversity. There's ethnic diversity faith diversity the backgrounds, you know, you have you know, you have your the guys that are doing thinking of AJ with the working out and in Tanner with with fashion and and then me as a therapist and socrates as an architect and it's like There's so so much. It's just really fun to be part of such a diverse group of guys who just really love to be together Yeah, because there's a really singular focus. Yes Can you imagine being 50 or 60 and being divorced and having to start over again? We have a lot of those guys in the audience and who follow us. Yeah, what advice or what would you say to someone who's? Dreams crashed and they're starting over again. What would you tell that guy? Yeah, because they're going through what I call the void It's the void right here, right? It's just this place of emptiness.  I have to have the relationship That's that's me finding out that I'm okay Who I am and so just like the order disorder reorder It's like you're in a space of disorder Coming you will find a reorder. You're going to find your soul And you're going to be able to reclaim your life So many of the guys that I work with guys that are in my group and stuff They're going through some really tough times and it's really awesome to be able to watch a man Build his core get back to a really solid place and to be able to then Reestablish his life on a totally different platform Yeah, there's a lot of subgroups here. We have this fraternity of excellence. Yeah They seem like their 20s and 30s seems like young fathers that kind of thing. Yeah, it's kind of who hunter draws in Yeah for sure. Yeah, yeah Who would you say is the best candidate for the 21 convention? It's a great question best It's hard because to be able to qualify it with best I just the first thing that came into my head was just any man who was curious Yeah, I think that's the thing and there's so many men have this thing within them They know that they long for something different that they want something I know that I can be a different man. I know that I can be stronger. I know that I can There's something more out there and I think that curiosity that there's something more that I want That's the that's the man that I would I would love to I like that. Yeah For curious man many people Have thought this was a Organization for people who just went through a breakup or for the young guy that wants to go out there and pick up girls And that kind of thing. Yeah, yeah, exactly for the man who was curious. Yeah, I See us going in a new direction marketing wise Curiosity yeah, so any man can attend a conference like this and benefit from it I'd believe so and that's why I love the diversity. There's so much you you're gonna hear so many different angles about what life's all About and what masculinity is about? Yeah Three years ago. I remember there was a gentleman here. He had tears in his eyes for the entire event And he came back The next year and the next year and I saw him here Hmm at this conference sitting up straight happy smiling not a tear within a thousand miles and I believe he might have just gone through a divorce or something like that at you know at that time happiness can be strong Confident and he was very fragile when he came here. Yeah, I remember during a Q&A period he couldn't even get a sentence out and Looking at him now.  Yeah Absolutely Where do you feel that your message is the most? welcomed What do you think I? Haven't felt like my message is not welcomed You know, I know there's a lot of things that have gone on social media and everything But everybody I work with my colleagues at my home office the people Everybody's like you're doing good work. This is amazing. Yeah, what you're doing with my My writing I want to create I I would love to create a thing like Hunter has that has an online community as well just haven't done that yet, but So far it's like this is obviously to a lot of people something that is very much needed Women long for strong men. You see it all the time and And I think that's kind of Yeah, I haven't I haven't gotten the negative feedback people are like this is amazing.  I am a writer I have so what I've done is I've I do men's groups So I've taken the material that I've developed over the years With my groups and the men that go through the curriculum that I have and I've put it into books So I have four books plan to have five And so, yeah, so I have these all available on Amazon and Yeah, I really enjoy writing and We'll see where this all goes But yeah, I want to develop more things to make have more a little bit more influence in the world of men Yeah, yeah all in the links for all of these books will be on this video as well Okay, great absolutely, and then of course this video will also appear on my YouTube channel and to my audience, which is not a It's not a male strictly male audience, and it's not a red pill audience. It's a it's really a very diverse audience I probably bring The most diversity to this group. Yeah than anyone at this point And I want to introduce this work to my people as well Do you have a favorite I know asking what is your favorite book is like asking what is your favorite kid kid? Yeah, right? Do you have one that just oh? You just bled all over the typewriter now. There's there's my age right there when I say typewriter All right, and that's what I was using George. Were you really? No gosh man, um All right, so That was tough man gotcha, so I know I know totally the Okay, so here All right the first one Awaking the internal is probably the most profound shift a man can take okay moving from External reference to internal reference living to what everybody else expects and what it's gonna make everybody else happy the External to internal what's gonna make me happy? What do I want? Where do I want my life to go? What's important to me? Easily the most profound shift the other one the other ones are important too, but my favorite the one I bled is Masculinity okay mastering the one you actually have up on the front here So the that feels like it's the one that if there if there's something that's going to really kind of I Feel like that's That kind of answer is a question. This this is the one here Are these sequential do people need to read these in order they help if they do that they are in order? They're pillar one two three and four, and then I'll have pillar five later is about relationships, and I haven't written it yet It's brewing trust me. It's all in here. Yeah, so yeah, it is sequential. That could be your next speech here or something Absolutely, yeah, it's all sequential. It's kind of you can move through this process Just like the men in my groups move through this process. So for sure give us a little preview of that relationship book Let's see Without revealing too much. We want to keep the mystery. I know I know I was just gosh It's not coming to me. There was a piece that I was just talking to with Socrates Why is that?  That's a really significant part of it and So Gosh now I lost it. Sorry, it's like too much pressure on On the spot on the spot. So relationships do help us mature. Absolutely Now there's one other thing that was even better than that. I can't think of it Sorry, it's not coming to me right now. It do you think that would go? Well, let's say with the patriarch convention Will it be ready by May? Okay, no, so maybe by next October maybe maybe maybe it all depends on how cuz I'm got so many irons in the fire Yeah, yeah Yeah, for sure. We'll see I almost see this being like a almost like a PBS series Well, I'm wanting to make it into online courses and online You know so that the guys in the online community will be able to kind of use this and go through it and move through all the material and Kind of and it's all about just gaining really strong internal strength and becoming a really man a man Who really knows who he is with the strong internal identity with just understanding masculinity and understanding how that is such a Blessing to the world It's not toxic. Yeah, it's beautiful. Yeah All right, so there's a man out there who's unsure of himself.  Okay. I would say you are good. You have a good heart Now all the stuff you thought about being something's wrong with me or or I I'm not good enough not true You are good enough you have a a lot to offer the world and don't believe all the negative things That have been told you about how you're toxic and how something's wrong with you but start seeking have that curiosity to really seek The really good voices that are speaking to you About how to create a really strong sense of self within you and go for it. It's out there You may not have all the answers But what's important is having the questions the curiosity a conversation with Ken Curry. Thank you Thanks, George appreciate it. What was your experience so far with the 21 convention? Oh I was dead. I was dead professional all across the board really good energy with a lot of people and I just like it's very positive Positive direction this George. This is a this has been a first-class event. It's fantastic. You guys were in a really tight ship I've been to a lot of conventions over the course of my business career And I can tell when things are well run and when things aren't and this is a very well-run operation I was very impressed. It's pretty incredible to see where Anthony's brought it Especially from last year which is my first year here and to see the upgrades he's made it's been incredible I've got my notebook and with every speaker. I've written down about two or three lines under each of the speakers of just Just the key prime stuff that I got that's good. That's great. It's very surreal man. I'm really enjoying it I'm happy to live in such an era Where such a thing like this is possible I have never seen a group of guys like this a group of 200 men We're focused squared away and working on their values just never met a bigger group of wonderful guys It's kind of neat because I've been to a fair amount of conventions in my day But you never see one where the guys like uh here you can just see ed latimore talking to tanner about boxing Yeah, you just sit down and then you tell your boxing experiences. Everybody's kind of pinging off each other. It's yeah It has been fantastic And it's been four days of guys all on the same page working in the same direction fascinating meaning some of the people Hearing their stories. Yep. You got people traveling from other parts of the world to come here just to see Yeah, some of the speakers. That's amazing. The thing is impressed me. Everybody cares very serious. Yeah, they're taking it You know close to their heart A great convention. Thanks, george


All right, so we have our expert panel here for managing Kubernetes at scale. And take it away, folks. Hey, everyone. I'm just going to do a quick intro of myself and then I'll let Naveen and Lisa do their own intros. And then we can just start answering any questions from the chat. So my name is Taylor. I've been a Red Hat just over two years. I work on the SRE platform team, which basically means that I'm on the team that manages OpenShift Dedicated, which is our Kubernetes as a service offering at Red Hat. Day to day, mostly I'm writing code that personally I work on the part that provisions AWS accounts and make sure that they're ready for new clusters to be installed into. But we also do rotations, on-call, all that kind of stuff, making sure that everything's running smoothly for the customers. Naveen? Sure. Hi, Naveen Malik. I'm also on the SRE platform team in service delivery. I've been with Red Hat for a bit over 12 years now on this team for, well, been over two years now. So I'm one of the team leads working with all our teams to make sure that OpenShift Dedicated continues to function and stays abreast of changes coming in OpenShift upstream. So pretty exciting place. Lisa? Thanks. So my name is Lisa Sealy. I'm a senior SRE also on the OpenShift Dedicated platform team. These two chokers are my colleagues. I've been at Red Hat exactly two years today. And I'm a functional team lead and two basically the same thing as everyone else. So yeah, we're basically here just to answer any questions you might have, discuss about what it means to manage Kubernetes at scale. We have something like 100 or so that we manage and hopefully that grows soon. But there's not, you know, the goal is that we have a minimal amount of engineers to manage the maximum amount of clusters. So that's what we're going to talk about. So yeah, anyone has any questions, feel free to drop in the chat and we can take it. Chat is quiet. Taylor, did you prepare any pre-canned questions for us? Wesley gave us one, which was why are tomatoes not fruit? So that was, I don't know if that's relevant, but it's an easy question. I mean, I think it'd be interesting to talk a little bit about like our how we manage configuration across all our clusters because I'm sure a lot of people using Ansible and Chef and things like that. But here, we have a question. Should we start with horizontal versus vertically scaling strategy with respect to Kubernetes nodes? Sure. I mean, we can talk through some of the considerations there. When we're looking at size of the cluster, it's really onto the customer dependent on the workload that the customer is going to be running. So this gets into a little bit more about the individual use cases, but you can have some constraints with either horizontal or vertical scaling, horizontal scaling with the guidance with OpenShift, I think as you hit 25 worker nodes, you need to start looking at scaling up vertically your masters. So that's one thing to consider. Not exact science has to do with load on APS server at CD, etc. So in our experience in looking at these things, there's some fuzziness. Also, if you're looking only solely at vertical scaling, there may be some infrastructure constraints that you have to consider for your individual nodes. For example, how many PVs can you mount to a node? We run predominantly in AWS, we do have a GCP offering as well. A lot of our answers may be biased towards that space. AWS, there's a certain number of volumes that you can mount to nodes may depend on the instance type as well. So that would be something to consider. It's a limiting factor for when you might decide to scale for its own only as well as like application workloads. Are you running something that's going to need to be on one node like it's a single pod that needs a large amount of memory? We saw a customer cluster recently where there was a pod with 24 gig request on memory. So those are things you need to consider. I'm sure there's many more. Yeah, I was going to say on our backend, we had an issue for provisioning new clusters where there's only a certain amount of jobs that can run on one particular node. So we had an autoscaler that would add new nodes as new jobs came in. But we also worked on minimizing the requests of the particular services running on each node so that we could actually run more jobs per node as well. So there's like, there's kind of two considerations like Naveen was saying like, depends on your workload, it depends on how efficient you can make your service, what the service is actually doing. As far as automatically scaling it for our end on our end for the customers, we don't usually do that customers will have to buy new nodes and then they can add it to their particular cluster to add on or resizing their instances as well as a second add on type of thing or as I find it, understand it correctly. So the thing that we do automatically is we just, we have a basically a configuration that says we need X amount of instances at Y size. And then we have automation that will go and do all that for the customers cluster. So we don't really control the autoscaling other than we have this configuration and an operator that looks at that and says, okay, I'm going to go spin up X amount of instances for the customers cluster. So we have some magic in the back end that manages that so that we don't have to think about that for every single customer workload. While we're waiting for more questions, yeah, I think it would be interesting to talk about those kind of config to every cluster, how do we manage that? How do we RBAC for example or certificates? How do we deploy those? Because at least for me too, that's also like, it's fascinating to see how you can manage from one single, like one single template, all these different clusters.  So for our newer offering before, we had to take some serious considerations on how to make that more efficient and how to not make a change, one like a single change take four hours to land on a customer's cluster because that was a problem. So Lisa and Naveen, do you have any insight into how we approach that for V4? I think the adoption of the operator pattern has been a game changer for us really, is that in order to scale and to have a small number of engineers manage a large number of clusters, we need to leverage software to do that. And the operator pattern lets us do that at scale. So as you were saying, Taylor, instead of having to have a bunch of Ansible running in a loop, which can take some time because at that time it was running through things that it didn't need to, we can pinpoint the changes by shipping customer resources out to a cluster. And now the problem we have to solve is how do we efficiently distribute small chunks of ammo out to a cluster so that these operators can do the thing that we need them to do in a very targeted fashion. And for me, I think that's that's been ultimately the game changer. Okay, so Kirk asks, you mentioned you manage over clusters and that number is growing. What would you say is the biggest challenge with running that many clusters? Would it be more of a technical problem or process problem? My personal opinion, I'm not going to speak for my whole team. I think we have more process problems than technical problems. The technical problems, yeah, they crop up, you know, like I said, the back end, like provisioning a number of clusters at the same time that becomes a problem. But my view on it is it's more about how do we manage those issues? How do we automate fixing things in the, you know, if we see the same alert over and over again for the same problem? How do we automate that? That is a lot harder, I think, than just I can implement the code. But how do I actually put it in a way that's going to prevent this from happening in the future? And how do I communicate that to my team so everyone's in the know of that? That's a lot harder, in my opinion, than the technical problems. But again, that's my perspective. So Naveena's team might have a completely different perspective. So I'll let you two answer it as well. Yeah, I mean, adding into that, there's the utilization of the platforms that are provisioned for customers and some of the surprising things that customers try and do in the odd ways that they then cause issues either, you know, surfacing in, you know, requests for manual changes on a cluster is which for OpenShift dedicated on version four is predominantly a no answer to how do you continue to scale like everything needs to be automated. We need to have change management for all things. So that's for myself, if I'm a mess in the question of what are some of the areas where I am concerned or see pain points, it's on how do we continue to grow the offer and scale it without having to bring in additional headcount in order to manage it. And we can do a lot around process around tooling around the configuration management through operators at least Lisa mentioned. But the car vol is the customer and what they do. Sometimes it's just like really You want to add anything, Lisa? I agree with you Taylor is that it's the process and the information sharing Kubernetes and OpenShift are such wide problem spaces where there's so many things that you need to know that sharing information among the team and having that scale to all of the clusters that we manage in responding in a reasonable time is just such an insurmountable problem because in a world where we're treating our clusters largely the same, if we have a problem with one cluster, we're likely to see that in another cluster as well. So how we do our configuration, how we do our deployments, how we manage the load and the scaling of what we're doing from a process point of view is really to me the most challenging. Can I mute it Taylor? I press that button. Okay, I'm going to go on to the next question. Thank you Kirk for the question that I think that is probably our bigot the crux of our team is like how do we scale clusters but not headcount. So I think that was a great question. Can you speak about Argo CD, Tecton and OpenShift pipelines? What are those and how are those used if you use them? So my understanding is that they are like GitOps and CI CD pipelines for Kubernetes clusters. We on our team we mostly use the operator pattern, we kind of roll our own operators to manage our infrastructure and our configuration. So we don't use these directly on our team but I think they would accomplish a similar thing to my understanding. So I actually have a slight question. So Tecton and I'm pretty sure Argo CD is going to fit the same description but Tecton is a cloud native or Kubernetes native continuous integration continuous delivery build system similar to Jenkins, Jenkins SAC, things like that. It's driven by custom resources inside the cluster. My understanding is that OpenShift pipelines is built on top of Tecton. We are not that team so I can't speak too much to them. I have used Tecton in my own personal stuff and I have found it to be really, really, really cool. If you haven't had a chance to check it out, definitely check it out. It's just really cool. And as alluded to, we do have our CI CD pipelines that are just not built on these tools. I would imagine that what we do could be likely implemented on these tools as well. You're looking at how do you get your source built pushed out to a registry somewhere for it to be available. How do you then get the configuration of custom resources, your deployments, whatever deployed into target cluster, clusters. If it's not like being run on the cluster that needs it. So all those types of capabilities I would imagine being there. I personally have had probably the least experience with these that the other two folks here have. Just speaking in generalities. Yeah, we use a tool called Hive to manage the cluster configurations and the Hive operator that's on GitHub. That's kind of our SRE service delivery managed. I don't know. It's a tool that installs OpenShift clusters but we also use some of their CRDs to sync configuration. So it achieves the same goal but sort of. Okay. I'm going to move on to the next question. A lot of new tools are using the sidecar concept. When scaling, do these multiple sidecar containers present some challenges? It depends on what you mean by scaling. If you're scaling an individual cluster, then the more containers you have running inside of the pod, the more resources that pod is going to need to be scheduled. Inside the pod spec for each container, you can have the resource requests. If you have something injecting sidecars, then that injector should be aware of the thing that it's injecting and the resources that it requests or requires and provide those appropriately so that the scheduler can find the right node to accommodate it. So you don't want to have, for example, if you have like an authentication proxy, that is generally pretty lightweight. So you don't need that to request four cores and 10 gigabytes of memory. So hopefully that injector would clamp it down some in your pod spec. If you're running your own sidecar, it's with multiple containers inside of a spec would do the same thing. If you're scaling in terms of the number of clusters, then the sidecar concept can give you a lot of power in terms of that automation that we all need to scale large. So it can be out, but you also need to be aware of the resources and I'd like to welcome my co-pad to the session. Co-presenter. Oh, you see your sidecar? Yeah, I would say the sidecar probably adds just more complexity to the workloads rather than like a specific, it's not necessarily add specific scaling issues because Kubernetes had to handle pods, but I think it adds more complexity in terms of like figuring out what's the issue if there is a scaling issue with your pod and debugging your service mesh is notoriously complicated because it basically adds sidecars to every single pod to do proxying for everything and that complexity can just get in the way when Kubernetes tries to do its thing. So I don't think it's inherently a bigger challenge, but I think the complexity doesn't help. Yeah, there's two things that if you're seeing concerns with sidecars or size of pods in general that you might want to look at. They're in tech preview and OpenShift 4.5. One is a vertical pod auto scaler, so as your pod needs more resources, having the ability to adjust the requests and limits on that, that's something to consider. And then there's the de-scheduler. So if you've ever worked with Kubernetes platforms, like scheduling is key, like how do you get your workload anywhere? Schedule it. If you need to do something like redistribute workloads, like if you're vertically scaling pods, like if you may hit a point where a node is overtaxed, having the ability to then move that workload by de-scheduling and allowing the scheduler to find some better place for it is something we're definitely keeping an eye on. Just to give an example how might leverage in the future when we provision in clusters, OCP doesn't on version 4, doesn't provide infrastructure nodes out of the box, so we add them. And what that means is there's some core components that get scheduled to workers that we would like to move to IMPRA. Right now, we don't have a great solution for that de-scheduler. It would be something where we could say, hey, if you see IMPRA and this workload is on worker de-scheduler, let it get re-scheduled to IMPRA. Thank you, Nureen and Lisa. Next question. I understand all of you are from the SRE team. What's a typical day like for you, i.e., what tools do you use? What are some of the common tasks, common challenges, etc.? On our team, we kind of have multiple roles. So we have a rotating primary and secondary, which primary, we have pager duty set up from each cluster, and then a Prometheus stack and alert manager stack on each cluster. So if something alerts on a cluster, the primary will get a page about it and respond to it. Secondary is mainly for being the interrupt catcher and interfacing with customer support tickets that our support team opens with us. And then everyone else kind of gets assigned a JIRA task. It's an agile development process. We have epics that we do, and it's usually go-based. All of our operators are go-based, and we have some other BASH and Python tools. So as far as common tasks, it depends on what role you are that day. Sometimes I'm just looking at pager duty, looking at alerts and responding to them or mediating things. Sometimes I'm just responding to tickets or investigating for a customer. Sometimes I'm running some go. That's for me.  It was once a quarter now. It's a great way to ensure that everybody on the team understands what's going on, and we all have an opportunity to be motivated to make the changes where necessary to help us scale the platform offering without having to scale people. I mean, I should caveat it. That means trying to focus more on engineering tasks over operational tasks, meaning not in having to invest additional people resources and operations. Like engineering, there's always features. There's this new stuff coming out to be a business unit. It's always asking for something new. It's never dull. I can give my couple cents. As a team lead, my common tools are Slack, Jira, VS Code, our video conferencing tools and command line, things like that. Click Taylor and Naveen and everyone else. I follow the rotation for primary and secondary. As a matter of fact, I'm secondary next week. Some of the common challenges, as I see it, is really information sharing. Except the SREP team is now 50 strong or more, I think. We're globally distributed. She means we have people in North America, in the Asia-Pacific region, and sharing information, something that I learned today, or created today, sharing that around the globe is the challenge. Sharing it with 50 people is the challenge.  It's more importantly, not keeping up the data than things that don't necessarily need to know about right away. As far as our tools and tech stack goes, we use, obviously, Kubernetes. We have intimate knowledge about Kubernetes and the different resources, plus the OpenShift specific stuff. Prometheus and Alert Manager that's what provides us our alerts and our metrics. They are good to know. We work with a logging stack, so EFK, mostly just debugging customer issues. But it's good to have knowledge of that. Git is how we manage everything. I would say think of us like software developers that also have alert setup, basically. It is not a real pager. It is an application that goes on your phone. And then you can manage it all on the web UI to page you. It just sends you a notification. It used to be a lot worse, I will say, too. I've been woken up at 2 and 3 AM when I was originally on the team, but now we have good global distribution. I don't get woken up as much anymore, which is great. Yeah, like Naveen said, I think the common tasks for ops are fine, but we really want to focus more on trying to minimize as many ops tasks as we need to do. That's our goal is to try and automate everything. The challenge is figuring out how to automate those things. Do we actually care about this alert, for example? How much noise do we have? How do we reduce that? How do we automate things? How do we make it so that we don't actually have to touch anything and that everything works perfectly? And balancing that against actually remediating the issues that we're finding, because usually we're drowning in tickets and drowning in alerts, at least when I'm on call. I don't know when you guys are on call, but when I'm on call, I'm drowning. So it's like striking a balance between we got to fix this issue for the customer now and also we want to fix this customer issue forever. Doing one is not necessarily doing the other. And I think, yeah, fill it a little bit on what you said, Taylor, about the interrupts in the middle of the night going away. We've made continual efforts to improve how we manage on call. A recent change, we took North America and split it into two shifts. So instead of a single eight hour shift for a week, we have now four and a half hour shifts split across East and West Coast. Weekend on call used to be from Friday to end of Sunday, you were on call. Now it's in eight hour, I think it's eight or 12 hour blocks, I can't remember. So that's something to also consider as you're scaling out, making sure your people don't break. Yeah, it's like, is there a balance between like process and engineering, as we talked about earlier? Technically, we can probably solve anything, but it's just about doing it the right way and dancing with all these other things and getting the right, you know, page or duty schedule so that I'm not woken up at 2 a.m. And I'm happy the next day. So what percentage of your time has spent writing codes, e.g. scripture and for code? I think this highly depends on the person within our team, but we aim for the Google SRE book standard. I don't think we actually hit it a ton. I really only do a lot of ops stuff when I'm on call, and I probably should do more. Some people do more ops stuff and don't write as much code. We have some specializations within the team, but it really, we probably do a lot more engineering or we have in the last year because we've been writing out this new process for v4 infrastructure, but now that we actually have had it settled, we're doing a lot more like remediation and ops fixes. So I don't know the exact percentage. Let me put it that way. It's probably not what it should be, but our team is working hard to balance that so that everyone on our team has equal time above and that we have good ops representation and people that know how to do that and also people that know how to write the right software or the right code and to manage it on that side. So it's kind of this weird mix of the two, nothing to add. That's the team and I don't write a lot of code very much these days, but the one I do. Yeah, I will say from a year ago till now I write less code, but I think that's more because I'm just in more meetings and I'm helping fix things more often, but I don't necessarily know if it's ops because I don't know. It's hard to explain. Well, it's a strange dichotomy, right, is that what we're really doing is we're writing software to perform operations tasks. And that is, to me, one of the things that sticks out about the SRE is when I try to explain the rule on a 10-second elevator ride, which is impossible, by the way, for many reasons, is I write software. I'm a software developer to do my operations, to do my systems administration. And some of that happens to be driving features. But the reason it's driving features is so that as a person with a system in background, I don't have to go in and configure the system by hand, how their software could do it. Awesome. I'm going to keep moving on. We're coming up on time. We have a few questions in the queue. Would you say you spend more time with customers or with the engineers who wrote the stuff your customers are running? I would say we, so the software the customers are running in the context of the upstream components that we deploy and manage, I would say we spend more time with those engineers than customers as a team. The story varies individual to individual. I know there are specific people in the team who spend more time, larger chunks of time with customers who work with large prospective customers, existing customers on capabilities of what we do, how things work. Those are always conversations that are happening, but for the broad team, I would say it's more of the engineering folks. And to go back to something Lisa said earlier, the communication being so important. That's something that we're continually also improving or striving to improve with our how we engage with and contribute to upstream components and making sure that what we're deploying, if we find an issue, we don't just like, well, we'll work around it and be done with it and we file bugs, we open conversations with the engineers if it's something that's time critical or going to be impacting a large swath of our customer clusters. So it's critical for us. And I think that's the perfect lead into the next question, which is how customized is your Prometheus slash alert manager versus the standard out of the box config? We have customized the OCP a bit, mostly the alert manager to say, you know, we want only alerts on X, Y and Z namespaces so that we're not getting alerts on customer namespaces and customer workloads. And we've added a few alerts that we've found over time that are either needed or are too noisy. We've tuned them, but there's been a recent effort that we are contributing back upstream with the monitoring team within Red Hat. So we customize it in the short term to contribute upstream in the long term. You can see what we use. We use configuration as a configuration management operator, CMO. That installs Prometheus and cluster monitoring operator, nice. That's on GitHub. And our rules that we deploy are also in GitHub under an edge cluster config. And the Page2D operator can configure the Prometheus operator, both interact with the stack as well. And they're both on GitHub. So yes, we try to do minimal customization. And when we do, we try to make sure that it's out there so people can see and that we can hopefully contribute it back upstream. Or if it's specific to us, like the routing stuff, we make that explicit in our public repo so that we can say, we're only monitoring these. We don't care about customer workloads in the sense that if they break it, we don't want to be alerted on it. So we have customized it just to look at things that we're responsible for to make sure that the cluster is up and running and that it's all healthy. That's great. We manage the platform. So the platform is our primary concern. And we want to make sure that the platform is healthy. And for us to be better equipped to do that, we focus on the platform related alerts. I think that's kind of the weird thing about a managed Kubernetes service is that it's managed in the sense that we just guarantee the platform is up. But it's up to the customer to know how Kubernetes works and how to architect their own workloads for Kubernetes to keep them running to a certain extent. We're just responsible for making Kubernetes available to you, which is interesting. Yeah, that's our team. Pat as a whole has courses and certifications to offer to help out. Yeah, our specific OpenShift dedicated service is that that's what that's for. OCP and Kubernetes in general, Red Hat has other offerings for that. Okay. What tool do you use provisioning the infra? So we use Hive as I talked about earlier, which is a basically core OS. They have the tectonic installer. And it's basically the Red Hat integration of that. And basically what all we do on the OpenShift dedicated team is we have operators that prepare a GCP or AWS account for the installation. And then we hand it to Hive, essentially, there's an API over all this that does all that handling back and forth. But we hand it to Hive and then Hive uses the prepared account and installs the infrastructure in it. So that piece of software runs it. And another team actually manages that piece of software in our bigger stack. So we actually interface with other teams to make sure that when you hit, you click on in the UI, create a cluster that it actually creates. But we rely on other teams and other pieces of software that we don't even really manage for that to happen. The short answer is Hive. Yeah, it's a team effort, as Taylor said, is the customer inputs into a tool called OCM. OCM talks to Hive, Hive talks to clusters and uses the installer. And once the cluster is installed, workload, like platform workloads get installed in it and become configured. And so it really is a team effort. If you take a look at OpenShift and how to install it, that basically the stuff at the end of the day, the infrastructure provisioning is managed by the installer, which is provided by OpenShift. And we're using IPI or installer provisioned infrastructure. As Taylor said, in the case where the customer doesn't bring their own account, we provide an account that's never been used before. And the installer reached out to everything other than the initial user that is needed to create all the various things that run the cluster. All right. How many times have customers broken something you need to fix anyway? Probably once a day, at least not. It's more often than it should be. Let me put it down. This is an interesting question. It is. How do you define broken and fix? I'll just give an example. So I was a dev comp in Bernou in January, and I got paged in the opening keynote because a customer had effectively broken their cluster by overloading the workers. And at that time, we hadn't been provisioning infrastructure nodes. So routers were down, and Prometheus was down, and Registry was down on this particular customer cluster. They're just using the cluster. They couldn't do anything else with the cluster at that time. That's just one of those examples where the platform was impacted by customer actions. And what do you do? Work with a customer. I figure out how can we get past this short-term solution? What do we need to do in order to protect our components from customer impacts like this, provisioning infrastructure nodes that are dedicated to running key workloads that then are not impacted when the customer hoses the workers? That was a really interesting example. But typically, their workloads don't impact our support of the platform, except for some very interesting education cases. As I mentioned earlier, sometimes customers just do weird things. I also think that sometimes customers are not fully aware or fully educated on Kubernetes, which is totally fine. And they ask us questions, or maybe our platform, they don't understand it. So they ask us, and it's like, well, we're not, we don't really do that, but we'll help you. LDAP, for example, customers setting up LDAP to log into their clusters. It's all automated for the customer. They just have to fill in these boxes. Sometimes they can't figure it out. Sometimes their LDAP server is unreachable, and we have to tell them that. So it's not often that it's necessarily the customer's broken something, but they can't figure it out. So we try to offer as much support there, even if it's not technically like the platform is down. So it's partly education as well, I think. Okay. If you get a chance to rethink your tooling, what new tools would you choose over existing tools if there are any such tools? There's this tool we use called Python. If we use Ruby instead. I knew it. I knew where this was going as soon as you said that. I have to bring up Hawke. That Hawke's correct works just fine. Thank you very much. No, but seriously, what I think that makes serious changes really, I think we've learned a lot since we've started down the path of using operators. And there's, I wouldn't say that I would change that we use operators, but there are a few places where we've implemented effectively services through operators where we might be hitting some scaling concerns. Basically, look at your boundary systems and some of the constraints you have when developing managing software. If you're running an operator that has custom resources, which defines something that needs to be done, that's an object in NCD. At some point, you're going to tip over NCD objects themselves at a certain limit and how much data they can have in an individual object, as well as just the platform can scale only so far. So that's one thing I would say we might reconsider. As well as the other aspect of that is it runs fine and great when it's a single pod on a single cluster, but now, oh, you need to run this on how many other clusters. That's one thing that I would say we think long and hard. Yeah, I would say it's not necessarily a specific tool, an architecture of how our tools work together. Just with the perspective we have now after a year of running this service, it'd be nice to be able to go back and be like, you're dumb, don't make this an operator, make this a REST API. This will be much better for you. But we have learned a ton about managing this stuff as a service, I think. And so it would be nice to retool. But it's also nice that we have those perspectives that we can say, okay, no, like given what we have, we can make it work, but we just have to be smarter about it. So there are tools I would replace, but I'm also glad that our team has the breadth of experience now post using them. Are there any common issues or gotchas you've seen in upgrading clusters? Yes, scheduling. It's always scheduling because we have customers that are globally distributed and we have team that's globally distributed. A customer in North America may not want their cluster created during North American business hours. So they may choose to have it upgraded in a minute hours, for example. If we have a lot of customers who are based in North America who want to happen at the same time, then that's a lot of additional load on those other teams in different regions. And if something should go wrong during an upgrade, because, well, each platform is the same, it's already open ship, they all should be more or less equal. The workloads on top of them that the customer chooses to use are wildly different. And that means that customers A workload is interacting in a different way than customer B's workload with the upgrade process, then that could cause problems. And scheduling it so that there are resources available to handle that is tricky, which deltales in with the number of combinations between what the customers can run in the platform. Yeah, I'd say from a technical point of view, there's been a few recent catches and we look at upgrades, open ship before the control plane's upgraded, and then the default configuration for worker upgrades is to update in one node at a time. We've hit cases with customers running a single pod with a pod disruption budget that didn't allow the pod to be turned off. So that node upgrade was indefinitely blocked. So that was a fun catcher that we've been building. I think our operator now that does upgrade, it's now does support a time limit on how long it'll be blocked for upgrading an individual node. But things like that come up periodically as well. We have an operator to do the upgrades instead of having initiate them at the anointed time. I expect them to be much smoother going forward in the happy case. Yeah, I would say they actually are much better than that. Now that we have this operator, but there are a few gotchas with the automation because sometimes the pod can't be can't be removed because it can't be scheduled on the new node. And there are just a few minor technical things like that that are just like how do you automate unknown errors? Or at least how do you how do you page someone properly for that? It's really hard to kind of like measure that signal and respond to it properly, even non-automated. So I think upgrades are always going to be hard no matter what though, like that's just kind of how Kubernetes is. OpenShift is great though. OpenShift is better than Kubernetes. So use Red Hat, you know. Anyway, I think that concludes our time. So thank you all very much for the questions. Karan, just put a breakout room for further discussion. So head there if you'd like to discuss anything we discussed here further. Yep. Thank you so much, folks, for answering the questions.