Hello everybody. We're still waiting on Justin, but he should be on his way over here. So let's get started here Again, welcome to flock it's really exciting to see people in person There are a lot more people here than there were 10 minutes ago. So I think that's a it's a good sign Hopefully some people on the stream as well possibly people here at the conference on the stream in their hotel rooms because they didn't want to actually get out of bed after the Pub trivia last night. We'll we'll see Hopefully more people be around soon But a couple housekeeping things first Do we have evening events coming up? There is a scavenger hunt and What else tonight and dinner Wait, I'll get that part on the mic, but yet So yeah, there's dinner at six and then a scavenger hunt There's a bus or two buses leaving from the front of this hotel at 5 30 sharp So be in front of the hotel at 5 15 or so to get everybody organized onto those buses We will leave people behind so take the city bus if that happens But also for the scavenger hunts. They wanted to have pre-arranged groups So go to the front desk and sign up for a group there We will put you into a group if you don't have you know Don't worry. We'll make everything. Okay. There's no no no pressure about finding the cool kids or anything like that We're all we're all cool kids here. So that'll be good There's a ghost tour tomorrow. You have to sign up for that as well And there's an early and a late one I assume that the ghosts come out more in the late tour than the early tour So depending on your level of ghost comfort you can decide whether you want to do that anyways Hi there I Am happy to present our keynote speaker who is also one of the main organizers of this conference Jen Madriaga who I first met at one of the I think one X foundation open source things a while ago and I Was very surprised to find somebody from marketing who was actually at one of our events and understood how Communities worked and open source and was just excited to talk about you know Fedora and actually had Linux the rel on her laptop amazing and So here she is Jen Madriaga who will be telling us very well prepared intro because I didn't prepare at all but Jen is very prepared That's that's that's assuming a lot Matthew. Hey everybody. Good morning Man, it was late night last night.  So Anyway, I know we're running a little bit late the keynote. I'm gonna try to be on time as much as possible even if we're Having a bit of a light start. I was gonna stand at podium, but apparently you can't see me if I'm Behind the laptop, so I'm gonna but I'm gonna be peeking in and out so I can Cheat and look at my notes. So Anyway, I'm here to talk to you about framing DEI globally It's a conversation that people have been having I think for a while for a variety of reasons and I Wanted to provide Some frameworks through which to view DEI As well as relate some of my personal Experiences around it because I think it's super important to have storytelling as a part of furthering DEI efforts So a little bit who I am. I think I know most people in the room, but So I've been working in open source for nearly nine years and Before working in tech industry. I was working in higher education administration nonprofits and arts organizations So I've got quite a bit of experience across the board and very different industries I'm someone who's got an international background. I've lived in numerous places Including abroad in the Philippines and France and in different regions of the United States Including Hawaii, California, Virginia, North Carolina, Rhode Island, and so I was born Hawaii I was in the Bay Area for a little bit Bebe Optiver to Virginia meets Virginia and in Providence, Rhode Island And now I am in Durham, North Carolina Which is 30 minutes north of Red Hat Tower if people know where that is And so it's part of my job. I travel pretty extensively. I've been to pretty much every region And I get to interact with a lot of different people, which is honestly one of the best parts of this role like it actually really satisfies my desire to learn new things and Meet new people because I just have a natural curiosity about the world And back in the day, I also did graduate work around issues of diversity and equity and inclusion and so DEI has evolved a lot in the time I did that kind of work and so I did a certificate in women's studies. It's actually not called women's studies anymore It's called gender sexuality and feminist studies and that was from Duke University in Durham, which is where I still live and then I also co-founded the Red Hat Asian Network and I currently serve as its chair But even though I'm chair the Asian Network, I am deeply interested in intersectionality because we don't just have one thing about us We have lots of different things about us, right? All of us are very multifaceted and that includes myself So before we get started I wanted to talk about the definitions over diversity equity inclusion and I also threw in justice in there And part of the reason I put this up is that I thought this was a really cool framing because it asks the series of questions And that's the way that I want people to approach DEI. There are no Absolutes. There's a lot of questioning. There's a lot of conversation. It is not a linear process, right? And I thought this was a really really nice framing And then, you know, most people probably have not seen the justice part, right? I feel like the justice part provides additional nuance to the diversity equity and inclusion piece, right? Because, you know, people think of diversity, like, okay, we just need a lot of different people in the room and it's solved But then, you know, you have to ask, well, who's not in the room and Then when you're in the room, are you allowed to be heard? Like, are your ideas allowed to be heard? And then If your idea is heard, is it going to be taken as seriously? Like, are people actually going to think that you have any credibility even if you choose to speak up? So so there's a series of steps in it and, you know, part of what I want to say about this journey, because it is a journey, is all of us are learning together So maybe you're learning, but you know what? I'm learning just as much as you are So I would never ever style myself as an expert. I can only say that here are my experiences. Here's the research Let's have a conversation and evaluate everything together because I was trained to be a researcher and a questioner and You know, I want to make sure that I have an understanding of where you are And I hope that I can express an understanding of where I'm at So I want to talk a little bit about diversity because people are like, well, what does diversity entail? And actually there was some research done by a person named Emy Edmondson out of Harvard And she actually is One of the very first people to write about something called psychological safety Has everyone heard of psychological safety? Raise your hand if you've heard of psychological safety Like we've been talking a lot about it. I'm going to talk a bit about what that means But before we go into psychological safety I want to talk about team diversity and so she identified four different types of team diversity along with Another research there Catherine roll off and a bunch of people are working in this field now So we're going to go through the different parts and then I want people to raise their hand if they think it applies to them So the first one is demographic Diversity So it means you've got a variety of people in your team. Maybe there's some disparity So maybe you've got a bunch of identity groups, you know, maybe they're from another country another part of the country Another race another culture different sexual orientation Stuff you know stuff that maybe this is your typical definition of diversity So does anyone belong to a team like that where you have people who kind of fit that definition? All right, so so a lot of you The second Type of diversity is expertise, so you've got a bunch of different skill sets You've got some people who may be senior you may have people who are junior You may have people who are really great at this one thing and other people are great at this another thing so Who has been in a group or is in a group where there are different levels of expertise where you have junior and senior Members or have been a part of team like that Okay, great All right, so this third one. I think this may actually apply to a lot of people geographically dispersed team members Who has to work with teams and other regions that are not where you live? Okay a bunch of you All right, and then the fourth one is disparity and so At Red Hat we like to think that we have a flat reporting line, but I work in marketing and so we're More of a traditional corporate structure where there are social norms of deference to authorities, so you know basically Usually in open source we say titles don't matter, but you know sometimes they do And so if you've got some sort of reporting structure where you've got a manager and people under them That would be basically that kind of framework So who is on a team where you've got someone above you and maybe people below you in terms of hierarchy Okay. All right, so it looks like the majority of you fit into One of these categories of diversity when it relates to your team or maybe all four I actually have all four so So yeah all sorts of diversity and so I think it's really important to frame what we mean by diversity And how it can apply in different situations So I have been doing this work diversity equity and inclusion And justice and other subsets within this world for a very very long time I don't want to tell you how long because I'll give away my age, but I've been doing this for decades and There's something that comes up in every situation that I've been in And I think this is universal And one of the things that I want people to take away is that DEI is not separate from your everyday experiences It's a part of how we experience the world even if you don't think You're experiencing the world in a way that involves that but some of my observations is That people have a very deep need to belong People are always asking What's what's my place in the world and can others understand my experience and Another thing that I've noticed is people need to be seen and validated You know people need to feel their contributions matter and not only do they matter, but I'd like to be recognized for the contributions that I make The other thing That's come up is can I bring my whole self into situations without fear or self-consciousness You know so an example would be like if people knew something about me, you know that I'm different in some way You're automatically operating from a place of fear and hypervigilance and that is not a fun place to be And so I think it's very very important for people to be able to be in an environment Where they can be their whole self without judgment and without fear And then another really fundamental thing that comes up time and time again, and it doesn't matter where your opinion is Around Dia is that people are really concerned about fairness People really focus on what's fair and how do we get to fairness right and to me at the way that I define fairness is You know, I want the same access to resources and support You know like if there's one group getting Access to support and resources I I'd like to think that I too would have the same access to those resources and support and then I also want access to opportunities I Want an opportunity to grow and to evolve Into something better than what I know Evolve into circumstances better than what I know So let's go back to psychological safety, right? So I talked about bringing the whole self to work or even to your everyday life. Maybe the grocery store You know, maybe while you're walking around your neighborhood So Psychological safety, you know, maybe you think of it in terms of being an individual But it relates to a setting or role Where people can express themselves, right? They can And that and in this particular instance work relevant thoughts and feelings, right? And psychological safety also includes in its definition Permission to make mistakes If you've made a mistake, can you raise your hand who's who's ever made a mistake if you haven't I want to know who you are I'd like I'd like to know who you are if you've never made a mistake, right? And so not only is there permission to make mistakes But there's permission to learn from those mistakes You know back in the day We used to have this sticker at Red Hat that said failed better. Does anyone remember that? I don't know if anyone ever saw that sticker. I actually have that in my laptop I loved it because that was like such a mind-blowing thing to see something like that in the workplace And I took it away because people are misinterpreting it and saying, okay, yeah that People are understanding what it means, but for me, I was like, this is great You know why because it means I'm allowed to make mistakes and learn from them and be better after it. That's how I interpreted it and so I Think everyone should have permission to fail better. I give you permission to fail better, you know so Don't be self-conscious if you make mistakes I think the key to is if you make a mistake What do you learn from it? How can you grow from it? But you know what we can't define psychological safety without talking about teams Because really psychological safety is Related and dependent on teams and why is that? Why would teams be important in Psychological safety hmm You know Part of it is a setting right like think about Maybe a past work experience Or a past school experience think about a teacher or a leader Think about the ones that made you feel afraid and judged and then think about the places where you felt safe Or you could laugh Where you could share things that were uncomfortable and know that you would be supported Have people been in both those environments? Have you been in an environment where it's like? You know, I'm afraid to speak up because I'm not sure what that means for me and have you been in an environment where it's Hey, this is great and with people that I really trust and I know that I can rely on them to have my back And actually, that's how I feel like when I'm at flock. So thank you community I always feel super supported in this community and the fedora community is one of my favorite communities for that reason Like I can nerd out and be my weird self Without judgment. I don't know that jackhammer sounds like it has judgment, but it's okay All right Another thing I want to talk about right if is if we talk about diversity by itself You know people assume falsely That if you create a diverse team Everything's solved Yay Hey, all of our issues are shot because we put a bunch of different people together And they're gonna work on a team and they're gonna solve everything because they're all different well Studies have shown that diverse teams often underperform relative to homogenous teams What you mean putting a bunch of different people together actually doesn't work To create an environment that's more productive Why is that? Part of the reason why it doesn't work is that there is no framework for communication You have to reimagine the world When you are working with people that are different from you So just think about it if you have a bunch of senior members on your team that have been doing Open source for 10 15 years and you have a newbie that comes in that doesn't know the terminology That knows how to access command line, but they're not sure where to start or where to go You have to interact with that person differently You can't interact with that person like they have 10 or 15 years of experience and if you did What does that bring out right It brings out a lot of fear and smallness because this person doesn't want to ask questions because They know that they're assumed to have a certain knowledge and skill set that they don't possess So we have to set expectations. We have to set frameworks for when we deal with team members that come in and as I said, there are different forms of diversity Right, so there are different things that you may have to do to address those types of diversity but the key thing is creating a communication framework into which all groups can work and understand right so Here are some things that you can do To ensure that your diverse team excels right So framing is super super important We have to be able to frame things as a way To provide opportunities for information sharing So if you've got that junior person on your team That's never been An open source and they but they want to be an open source You have to create a framing with meetings where it's opportunity to share information You know Information sharing is really really key right and then we want to frame differences as a source of value right, you know One of the things I loved when I came to red hat is the idea that the best idea wins and it Doesn't matter where it comes from doesn't matter if it comes from your intern or the CEO the best idea wins Right, and that's part of you know Like I think about like even in terms of my height because I'm really short and there are things I don't think about because I'm short so for instance when I'm on the airplane and I stand up on my seat I don't hit This part like I just stand there Versus like someone like Adam probably can't do that, but it's not part of my everyday experience So I don't think about stuff like that And then another thing that I think about is that like my refrigerators very high at home and my husband likes put things on the top Shelf and I don't ever see them. I can only see I level Right, and so we have to consider What are people experiencing right and you have to frame those differences, right? So one of the things I've asked my husband ideas that can you put stuff that I actually use on the lower shelf So that I can reach it and actually see it and not buy like five other items that are identical because I didn't realize it All that stuff is on the top shelf So anyway, so that says very important, you know, you want to you want to have conversations up frame those differences And then increase like I told people I have a natural curiosity About the world and I ask a lot of questions. I ask a lot open-ended questions and Sometimes listening is really really hard, you know, especially if we're already attached to an outcome or Attached to a certain expectation or attached to a belief system, you know You know, I may be at the mindset because I don't ever hit my head when I stand up on a plane That like I don't know what the big deal is why where people complain about not having enough space.  I can stand perfectly fine. You know, I've got clearance too You know But you know what I know That's not the experience of other people You know, I'm sure that folks who are Vertically challenged this way Have their own sets of issues, you know, maybe you can't be in a hotel room without having your feet hang over the bed You know, and it's impossible for you not to ever have your knees hit the seat in front of you Which is probably pretty excruciating On a long flight, right and so I want to ask questions like hey Adam, what's that like? Do you find that comfortable or do you wish that on other people? Yeah, he wishes on other people, okay, that's good to know. All right But part of these questions, right is you also are trying to find like shared ownership and causality And what do we mean by shared ownership and causality part of it is we're trying to Establish understanding of different things, right? So we're we're putting things in the pilots I I call it like recipe making so like if you're trying to create something As a group if you're in a project of some sort I always like to use the recipe analogy where like if you're wanting to bake a cake You can't bake a cake just without Would just flour or with just eggs or would just butter or with just sugar you need all those things To come together with a certain magic So that you can have a baked good at the end, and I love baked goods So who doesn't like it baked goods, right? And then we have to talk about bridging boundaries, right so People have a natural threshold for comfort because of their experiences, you know They may be only comfortable with You know this much space like personal space is another thing that I think is always really intriguing when I travel I think in the United States. We like to have an arms length As comfortable personal space But I've been in cultures where people are right on top of me and almost touching me And it's kind of like whoa, okay. There's something I have to understand here because this is their norm. It's not my norm and So I need to figure out where people's boundaries are right And so part of how you bridge boundaries again is without open-ended questions Like if we're working in a project everyone's got hopes and goals for that project, right? So if we're in a work setting a lot of those are assigned things that we're needing to do and accomplish And it may be that the various people on your team Have very different hopes and goals and you know what that's actually very important to know that So your senior person may have very different goals from your junior person and You need to have a recognition of what they are so those gaps can be bridged, right? And then you want to do an inventory of your resources and skills, right? So one thing like With a newer person is they may see things that you won't see because you're so used to doing things a certain way You may not see something that actually needs to be seen and someone less experienced or less familiar Can see those things because they're still learning And so those are the things you want to think about is that you know what just because this person does not have the same experience as you do They're still bringing something and then This part about concerns and obstacles. I think is super super important particularly if you are leading a team But this is a conversation that you also want to have as a team So people have an understanding of what's going on like what are people up against? What are you worried about? You know sometimes these are one-on-one conversations and sometimes these are conversations that you Want to have as a team, but it's very very important what people's obstacles are Like an actuality maybe would process but also You know psychologically, you know, maybe people are held back by fear They don't want to make a mistake because if they make a mistake then oh no the world ends, right? But the world really end So part of it is you have to put people's fears and perspectives and maybe you have to put your own fears and perspectives because usually The world won't end maybe your servers will go down for two months and that's terrible But you know the world's not really ending Or for some people might so I want to talk a little bit about inclusion and how you evaluate teams So this is I'm not going to read through all these questions, but I found this article by Chelsea Troy about how you evaluate team members contributions for our inclusive culture and I've already kind of gone through a version of this with my last slide But I thought this was really great, you know so part of what you want to look at in your team there Are they able to moderate? Their meetings are they able to make sure that people are Expressing their opinions and then if they're not, you know find a way to solicit them I mean there are people who are both extroverts introverts. There's some people that like to give verbal feedback or Written feedback and so it's really really important to provide different forms to To do meetings for feedback So same with the soliciting opinions attribution. Oh attribution is super important So one thing that I think That I already spoke about was the need for people to feel recognized, right? And in another really common experience is people Expressioning an ID and then someone taking credit for it, right? So one thing that I think that's really really important to realize is that it's important to Recognize a contribution given by someone And not appropriating it and if you're appropriating it, you know ask yourself why you're doing that Usually it's related to wanting to have power over someone and You know that actually doesn't really create a great team dynamic and what it does is it shuts people down So, you know, let's go ahead and get recognition to people because you know what it lifts you up to Because you're seen as someone who is a supporter as someone who is here to uplift and make people's lives better And honestly, you will get so much in return by doing that instead of being small and taking credit for something that you didn't do So, all right, I'm just gonna go through that. So because I want to be mindful of time So I want to talk about the power of the individual So we talk a lot about individuals particularly within American culture And I was talking to somebody about this yesterday, you know, because people always ask about like what can my company do? What can my organization do? But sometimes it's up to the individual. It's up to an individual exercising their own power Even when they're fearful. So one of my favorite quotes is Speak to truth even if your voice shakes and my voice Shakes on many occasions because I'm not a person That is naturally comfortable with putting myself out there. This is something that I've cultivated Over many years. This is not something that's comfortable for me. In fact, I was conditioned not to speak up My dad was always saying don't rock the boat. You don't want to draw attention to yourself But there are times we do have to to speak up And I'll talk a little bit about my work and creating the Asian Network at Red Hat in just a bit But what's important to realize that sometimes the greatest changes don't come from organizations themselves it comes from individuals Within organizations or me outside organizations. They're willing to put themselves out there to do the work without an expectation of compensation and Then it also involves relationship building because allies are such an important part of Making change happen like I can't make change happen By myself. I have to make sure that other people have buy-in to making things change and so Individuals can start those processes and then they can catalyze Things happening But then we also have to talk about leadership Because leadership does matter One thing about individuals creating great things is that if that individual leaves your organization, there's usually a void and so Your leadership It's necessary to implement structural changes. So as an example Within the United States We had to do a voting rights act for people to have equal access to voting and that's actually being challenged right now but in theory everyone always had the right to vote but in order to ensure legislation had to be enacted and Other words structures had to be changed because if you're looking to individuals or existing structures to make change Happen it's not going to happen you were going to have to enact change on a structural level and in this case you would do it through legislation also You need leadership because remember when I was talking about when you have diverse teams you need to have a structure for communication It's your leaders are going to make sure that that infrastructure gets created and So even the individuals are greatly important if you want lasting change in your organization You are going to have to have the buy-in of your leadership to make those things happen So let's talk about creating inclusive leaders, right? you know, so Again, I'm a fan of the Harvard Business Review. So you'll see that I get a lot of citations from articles And if you want a copy of my slide deck, I'm glad to give it to you because I have all these links embedded But when they quizzed leaders about being inclusive Inclusive leaders and we'll go through what makes it an inclusively or just a bit. They're 17% percent more likely to report that they're high-performing 20% more likely to say they make high-quality decisions and 29% more likely to report behaving collaboratively and Then it also Increases work attendance. So basically, you know, if you're really hyper focused on a productivity Aspect of DEI maybe not so much the human element There is a correlation that if you practice inclusion inclusivity that you will thrive so what are some examples of Strong inclusive leadership So when they surveyed people when they defined a strong inclusive leader or they said that strong inclusive leader is not afraid to share their personal weakness and An example of personal weakness may be You know, I know I'm good at this, but I am not good at this and I need your help I Need your help in making sure that I'm understanding things because this is not my area of strength So that would be an example of sharing a personal weakness The other is just learning about cultural differences and realizing that there are different styles and so Like I said cultural could even extend as someone being introvert or extrovert As well as people being from different regions different cultures or different countries But understanding, okay, here's a preferred way to communicate Here's something that needs to be highlighted You know and so that's super important that this leader leans into learning about those things and doesn't assume Things they actually want to learn about those different things and then also Acknowledging team members as individuals so, you know, one of my favorite stories I like to tell about when I came to Red Hat was The afternoon of the first day of new hire orientation we had a reception where your teams could come and Welcome you and I remember There is this gentleman with blue jeans a belt and a white button-down shirt. I'm like, who is that person? Why is this person here? And he was he's like hey, I'm here because I want to welcome you to Red Hat and it was Jim Whitehurst our CEO And I'll tell you I was like what kind of place am I working at Where the CEO comes to shake my hand and to welcome me and tell me That he's looking forward to me helping me the organization better I Mean, I still remember that story and I also remember the times that he would come to the booth Completely uninvited because he wanted to say hi to Hatter's and he would take selfies with all of us Man, those were really great days. So Jim Whitehurst wherever you are. We love you And then least inclusive behaviors Overpowering others, you know like getting your way like my way or the highway Displaying favoritism like I you know, I think this person can do it best I don't care about the rest of you and then discounting alternative views like my views the only one that matters because it's the right one You're like sure that there's nothing else that can be introduced to change your viewpoint And one thing also that's really important to know is that you know, we're a lot organizations or fans of surveys You know feedback forms But it's really important that feedback are distributed results rather than average results What do you think that is? Because if you average it out you hide Where the differences and disparities are in terms of perception it hides your problems And so be very suspicious. It's like hey 80% of people are really happy, you know You need to kind of dig down a little bit deeper. Is it is it really 80% or is it, you know, how is this calculated? Data can be manipulated any way you want So how can you be an inclusive leader? So it says inclusive leadership shadow I don't know if anyone is familiar with The psychologist young but he talks about the shadow side and All of us have that shadow side all of us are really convinced that we're really good at what we do But sometimes we're not and we have to ask people for their feedback And we have to be visible and vocal about all those things So one thing to remember is that inclusive leadership needs to be cultivated and conscious and When exists everyone does benefit from that and I know that we're running out of time So I'm gonna just blow through these last slides really quickly One thing I want to talk about is contact theory and this goes back to you know when people think oh if I have a diverse team Everything's solved if I put a bunch of different people together. It's solved like my DEI problems You know are gone and it's not true So one thing about like putting people together in a room There's a thing called contact theory and this one is specific to racial prejudice But if racial prejudice can only diminish when you're in a group with people is if you're under certain conditions And I think this also relates to Psychological safety that I talked about earlier, which is where everyone is sharing roughly equal status and belonging and working in a common purpose so if people are very different and they're in the same room just because They're together it does not mean that things will change the conditions have to be equitable The conditions have to be fair feel fair to the people who are in that room together And they also need to feel like they're part of a larger purpose So one of the things that I have to do on a regular basis is work with people with very different personalities different backgrounds For myself different philosophies of how to get things done But one thing that always remind folks when I'm doing a work project with them is like you know what in the end We want the same thing we want to accomplish something together and make it great What do you think needs to happen? for that to be the end result and That's Sometimes hard, but finding that larger purpose that you can agree on makes a huge difference so I'm gonna come back to the slide and so hopefully maybe you understand the justice part a little bit better You know the question that needs to happen for all of these things to be a reality in your organization And then I'm gonna just talk really quickly about Red Hat Asia Network And then I'm gonna let everyone go to their next session, but as I said I was a Co-founder for Red Hat Asia Network, and we were actually primarily concerned with Asian Americans Just wanted to shout out to our founding committee two of these folks are no longer at Red Hat One was laid off. He was a chairman's award winner Was working in the office of CTO and then another just left to go to another organization So there's me and there's Robin Chan my coach here and in Tesh Patel They're the only ones that left Joe to sigh and Laura Fu are no longer at Red Hat and actually executive sponsor just left three weeks ago So it's just a few of us are left who have done this effort So I just want to talk a little bit about why I did this organization Because I never talked about DEI at work. It's been a very long time and now I'm talking about it a lot But there are different things that happened one was COVID-19 and the racism that it brought out and Not only the racism But it also bought up a lot of old societal wounds for a lot Asian Americans about being othered and Not to be blamed for something was pretty painful And then another thing was that there were a lot of associates who are not being promoted into management Asian Americans are least likely group in the US be promoted to management within tech and then within tech It's Asian American women. Oh, that's me who are least likely to be promoted into management roles So one of the things that was really important to me was to challenge people's assumptions about belonging to a group So within the United States we have something called the model minority myth Which is that Asians are super successful and they don't need anybody's help And That's not true for a variety reasons first of all the term Asian American is a socio-political term That evolved in the 1960s It's an umbrella term for a wide variety of folks and the reason why it was created was to create a Way to ally together to help each other To find a more equitable place To find opportunities For equal access like other groups So I Just want to leave you with people being intersectional beings. So as I said, I'm intersectional I Identify as a woman of Filipino descent. I Am also neurodiverse. I have ADHD And I've lived in particular places and from the way that I am so Anyway, hopefully this is helpful Like I said, this conversation is not over and if you want to talk about a little bit more. I'm happy to talk with you But hopefully this provides some additional Lenses and other things for you to consider that maybe you haven't considered before So thank you for coming to my TED talk karaoke later Yeah, and stay in touch find me and Master on our Twitter or X or whatever it's called and then there's my LinkedIn as well as my email Okay, I'm really done now. Thank you. I think we can go to break now by the way Our session I So let's do this Well, we represent the four ninths of fesco so even we don't have quorum today And And normally we would cancel the meeting but let's let's I know Let's try for a bit So my name is Bishak. I work on system D and I have been in Fesco for Six years now I think Yeah My name is Neil. I've been in I do lots of stuff in Fedora and other places and things like that And I think I've been in Fesco. I think I first got a fesco at the start of the pandemic. So I've only lived in pandemic era I'm Kevin Finzi or Narek and I've been in Fesco since it was the Fedora extras steering committee That's a name I haven't heard in forever and I am David Cantrell. I have been on Fesco I guess also during the entirety of the pandemic. This is actually the first time We've met in person So that's that's kind of interesting to think about I've been in Fedora for a Long time I work I work on software. Yeah We do things we've been around So, yeah, I mean meet your fesco the way these normally go is we just kind of open it up to questions And you know if however technical you want to get or non technical we can talk about anything really Check if there are any questions online. I guess that's a really good idea So one thing that I That we tried is moving the discussion of Fedora changes to discourse discussion Fedora project org and I Think that the results were mixed We started with some very tough ones, so we are overloaded by the number of comments Personally For me email is just more comfortable easier to get around in and more familiar more efficient but On the other hand We had more comments from People from new people and people from outside of the community or fringes of the community Which was well Changing in some ways, but also good in other ways so I Don't think that it's at least for myself. I cannot say that did this move was Clearly good or clearly bad. I can see Yeah See it both ways I hope to That with other proposals which are less contentious we will have a smoother workflow and then maybe It will work better and And all that's that's that's how I view it. I was wondering how other people see it My brain melted during that discussion Yeah, well, no, I have I have some thoughts on it like the the I'm always in favor of making it easier for people to To provide constructive and valuable feedback and and input into the thing that they're using and building upon and and In being invested in right like being Enabling more participation is always great the flip side of it though has been The this particular this particular discussion wound up feeding itself into a very deconstructive destructive frenzy is I think the the easiest way I could describe it and It got amplified and re-amplified and fed on itself and Because The It's a it's the trade-off of you know, how far you want to go and what you know how much how much you want to have input from and I don't know if we figured out a good balance for this with putting it into discussion FPO I my my Personal opinion about this is I have some reservations about continuing to do this, but I'm also willing to see how some of the others go before making a Full judgment on it. I mean, I'm glad we're trying the experiment and seeing how that's going to work out and seeing What kind of participatory effects and what kind of quality of discussion we get? it gives us a good idea of Whether this is something we want to to continue down or not, but You know, I guess we'll see Yeah, my my observation has been that You know, we we wanted a way to get more input on change proposals That's the core item that we work by on fesco. You've probably seen them before There's a process we discuss it. We vote on it. That's how changes get into fedora On the mailing list side, which I will say I'm more comfortable with that only because I've been doing this for a long time The mailing list side it felt like or at least over the past couple of years Change proposals would only get comments by roughly the same five to ten people I don't know if that's because they were not as visible or that they People just felt like they were not You know welcome to comment on it or it was kind of a thing They couldn't participate in so Expanding that getting that outreach through discourse has been good because at least it makes our discussion on fesco easier because we Have input from more of the community, you know, what do people actually want to do? Is this something that people want? so just having more data points is useful, but I See the the the discourse side is having an opposite effect So the people that were comfortable on on the mailing list are now uncomfortable on discourse We can pick one set of people we want to hear from or or the other and and we can't get both but the answer isn't to Post it in both places. We need to come up with a better solution I do think that when we we tried the discourse method We did not really do a good job of explaining what the change proposal process is to people who had Largely ignored it. So I I observed a lot of Community members thinking that these were more of Announcements of decisions that had been made as opposed to discussions of changes So I think we could have done better at explaining what this was going to be and what we expected everyone to do So that was I think kind of new for everyone. I don't know if you guys like There was definitely like a lot of people said, you know, this is a fork on conclusion And then and I think there's there's a couple of reasons for that that at least I observe is like one of them was Are the way that that changes have landed in fedora over the last like, you know, decade That I've been observing this is largely been Uncontroversial super easy they go through there's the feet the feedback is discussed and things are resolved before they get to us So it starts looking like whenever they get to the Fesca level that they're quote unquote rubber stamp But the reason that they get to the point when they reach us that we're all comfortable with this because we're also participating in these discussions and people are resolving our concerns way before we get to that first meeting and so people have this Conception of we rubber stamp things when we actually don't And then on top of that the way that the news reports on fedora's changes process Like so if you look at some of the pharaonics and others There is a very very Important reason a very big reason why we now have this banner that says this is a proposal in the wiki pages Is because there was a number of incidents where? Volatile changes were being proposed that people assumed were going to just happen Some of them didn't actually happen But they didn't know that and they reported on it and misreported and then there's this like View that a change proposal isn't in fact a proposal But like the equivalent of like what a proprietary vendor does it says hey, we've done this thing your You got to just deal with it, right? Like that's that's the I think that's the two big parts of it Yeah, I think there's some some really good lessons learned from this this discussion that we had on that controversial change and I think that's one of the big Outtakes from it is that people don't understand the process and we need some way to tell them This is a proposal. You're supposed to give feedback. It's not done yet. It's still under process Also, I think there's a lot of Associated things around that like people didn't realize a lot of people thought with this particular pose proposal Oh, it's a team in red hat So red hat proposes things and then people just approve them Well, no anybody can propose something if you have something that you want to do and you willing to do the work And you're willing to write it up and get it through the process. That's fine. Anyone can do it but I think a lot of people didn't know the process and Perhaps were pointed at the discussion from a context like oh here You should say no to this and they just would go and say no and not actually read through the the proposal So I think there's lessons learned from it. I think I'm also on the mixed case I think we got a lot more input from people on the outside in the community and that's great I think expectations were bad. We learned some lessons Hopefully those will feed into this and make it actually better. I think it's still worth doing I think it's a good experiment and we don't have enough data points really yet. This is kind of a Crazy outlier this I hope it's a crazy outlier. I don't want I don't want discussions to be like this all the time How do we think the nano default editor proposal would have gone? So I think it would have actually gone the opposite of the mailing list you bring that up in and in all seriousness I think if we had let in in in theory crafting land that we had the nano discussion on Discourse rather than on the mailing list It would have been completely the opposite way Because the people that are on discourse are the complete different set of people Then what we've been interacting with on the mailing list a lot of them are either newbies or first timers or their Thinskin thin their first into Linux and and their their points of accessibility into the terminal and all this other stuff They would have appreciated the change more than a lot of the people that have been old school have been battered by the You know the old unixes and whatever's and they've they've lived by the scars of their heads and vies And so I think it would have actually gone the other way and it would have been contentious for a different reason Because you'd have some of those old-timers complaining that we just did something for you know Well, I think we've actually had some of those complaints during the thing about like we're we're Simplifying or or making it like we're not making it less. Yes. We were making it too easy and stuff like that Oh, yeah, that definitely comes up too often as people people think because it was hard for them We should keep it hard for everyone like I want everything to be easier than what I had it. Well, yeah I mean, that's kind of what we want to do so Comments from the audience please So we've robbed people of the learning experience of how to get out of them that I mean that's a good point and you know people people brought that up and you know, I It's kind of funny all the points that came up specifically on the the nano change proposal and I don't use nano I used them for like 25 years and and then changed to to emacs and Actually became way more productive, but that doesn't matter. I mean the editor is everyone's choice, right? And you didn't just slip that in just because right yeah, yeah But I I think I am one conversation I brought up like you know guys like gentu which is you know if you really just have a lot of time to burn and you want to Set up a Linux system you can't but their default editor. I think was was nano. Yeah, so you know It's it's viewed as kind of a high-tech, you know end user Sysad men distro and it's using nano. So, you know, whatever and another counter another counterpoint to this particular like you want to keep it hard so everybody learns the hard stuff is like I have skills for how to Screw around with DOS systems like I started with MS-DOS and I use DOS editor I screwed around with config dot sys and auto exec dot bat How many of y'all after have to touch one of those these days? I promise you almost none of you do the unless you're like doing retro stuff for the fun of it like I do sometimes so Like those kind like you don't need to keep things to be hard to keep it fun and to keep people learning You give them new interesting things to learn and grow from and there's new hard things for them too also Yeah, the hard things don't ever completely go away. They just change This thing is like nice to hear but Is I actually don't mind about either of them nano in whatever, but I would love to hear Your inside in the recent decision about the NF5. So That's the controversy. So put it on the stage and I especially curious Like yeah, it happened to revert But why it was reverted in in row hide especially why not to foster in in row hide and continue Testing development there. Yeah, so I think I Said this multiple times. I think that I'm very proud of the DNF team for like looking at this and deciding that it wasn't ready You know, most of the time people who invest so much time and energy and work into something And there's a deadline and they're not quite meeting the deadline they'll try and get it in anyway, but I think they did a really smart thing and a really a really good thing to Decide to pull back from there and I think the the question as to whether leaving it on in rawhide to get more testing I mean, it's still there people can use it. It's just not going to be default But we don't want rawhide to be used as a just a test bed where It's not actually intended to go into the next release We want it to be a integration point where people are actually testing the thing that will you know go to the next release So I I think it's good to revert it there until it's ready to to actually go in I don't know how much more you really much more testing would go on there It's hard to say but there's a lot of the stuff like the API wasn't stable until fairly recently as well And so there's going to be a lot of stuff that's porting over to it, but it still can be tested. It's just not default so Yeah, I'll add that the DNF team is the team that I work on as well and you know, it was I guess it was Disappointing but really the the correct decision it Just the feedback in general on DNF 5 and the current state leading up to Fedora 39 There was too much that was unfinished in my opinion I DNF is is one of those components in Fedora that we absolutely need to work and Since we already have one that that is implemented to move to one that that loses core Functionality that we have documented that we tell users they can rely on Just it it just felt incomplete. So it it needs a little more runway to to sort of Complete that and just this morning. I was talking about test days Implementing more DNF 5 test days We I think we came up with three that we need to do But the Festco ticket where we were talking about implementing the contingency plan to revert it There there were a lot of things that were pointed out there and to me I feel like that was a kind of a rare situation for us to get that much Feedback which which to me shows that people were trying it But a lot of stuff just didn't work yet and and it just I you know I like like Kevin said the the DNF team just saying like yeah, it's not ready We should we should pull it back and and even still Not just bouncing into the next release Which is another common thing too like we need to rethink our approach here and maybe it won't be ready for for two releases Yeah, and as someone who's you know I have a connection with the DNF team as well because I've been a community contributor For almost eight years now in in the various, you know RPM systems a software management group projects the The the DNF 5 work You know when we started when it started getting rolled out and people started using it like it was very very clear that The the use case prioritization There was a mismatch between what Fedora needed and what community needed it and what the team thought that needed to to be able to make it happen and We were encountering pain points and issues and the community was bringing them actually fairly promptly And that was that was also unusual usually We don't get anything for until like the last minute and things were just you know And they were being very very responsive in addressing the concerns and working through them But it's just when you get a huge pile of them and there's just only so much work You can do it once and you've got to qualify all that you got to make it work And and like David said it's the core of our system We have to make sure that it's actually solid to the point of like not breaking it like the one of the Reasons we went from yum to DNF was to massively increase the reliability of Doing the core thing that we need the distribution to do which is managed software and we don't want to screw that up in this transition And so I'm really happy that we're taking a step back to take that next step forward Justin go if you remember during the the yum to DNF switch You know that was actually rolled out it not not it wasn't the same state the DNF 5 is but it was a rolled out a Little prematurely as well, and there was core functionality of yum That was now missing from DNF and that caused a lot of problems now We got over them. It was fixed and you know DNF is great now, but We don't want to repeat those mistakes exactly Yeah, that I Was on the anaconda team at that that change point and one thing I remember is During that switch over we would we would have a bug report for yum. We would file a bug report That team would would fix it but fix it in DNF and Not yum so it was like we were starting to have the two drift apart And we just had eventually had to make that cut over and yeah We got we got past that but it was it was bumpy it was you know They wanted to say like look the time for DNF is now It is fundamentally different. I remember one of the the the big difficult things with yum and DNF At that time was Installation if you had yum as the back end in the anaconda you would get a different Resolved set of packages to install versus DNF because there were different depth solvers And that it's like well Okay, I guess we're just gonna have that now maybe and yeah So that's long since past but it's it is important to remember that so that we don't end up in the same kind of situation one that I I Was disappointed that is not fully implemented yet as the system upgrade capability that we rely on now in DNF That's not in DNF 5 and I was like well, I mean I use that Like that's got to be there, you know, so little things like that, you know, we definitely need to get Is my question is so do we reverting this stuff like do we have somewhere the list of things Issues which needs to be solved and not just in DNF, but with the integration with other tools Fedora review Discover other stuff like which needs to be solved. Oh my mark is done. That's that's my project So Which needs to be done as so we track it and that surprised one year later again Because I ask yeah, I reckon when I check was the status of the other plugins as well Not just copper plug in. Oh, it's tracking somewhere inside our GitHub like but it's not visible outside for the people who are not willing to dive into the bunch of Issues, etc. We have Yes, we have that information. It's incomplete right now because it's mostly from the first DNF 5 test day that we did but it does have a lot of those results What what did not work? What were we expecting bug reports were filed? They've been prioritized in my view They've been prioritized differently than what Fedora was expecting. So we're at a point now where we since we've reverted this in in Rawhide We need to go back as the DNF team and look at all this and say okay This is how we need to prioritize this work and yeah, so that information existed. It's coming together So there was some additional information on that in the Fesco ticket to people were saying these things are blockers And then you know some of them weren't some of them are this is implemented or it's different or whatever But some of them definitely were like system upgrade is not there and you know things like that Well, and you mentioned Fedora review, but also all the integration with Fedora infrastructure And like I did a lot of the early porting work for moving from young to DNF for all for a lot of the relinch scripts and things like that and I literally couldn't start figuring out how to port things from DNF 4 to 5 until like two weeks two or three weeks ago And that's really not enough lead time to be able to make sure everything works And the API wasn't stable until recently so right and it's not like I wasn't aware that they were working on it and think about it But like going through the effort of making all these changes, especially as a volunteer and spending that time and and not having documentation and not having Understanding of the expectations with the API interfaces and stuff like that's it's a real chore to go through all that stuff And like I'm not going to ask any volunteer to to do all that work when they don't know what they are supposed to know and You know once we like that's that piece of information is something we Kind of have started to get and I'm kind of and I'm hoping that that gets fleshed out more in the intervening months so that we can Actually start, you know giving that feedback from the API side because I know that the test days have given a lot of user feedback But we haven't done a whole lot of producer feedback Which is also important for being able to switch over to it because I don't want to repeat of where it took us Five years before we could start using rich dependencies because nobody could figure out how to port like the Bodey and other stuff to use DNF instead like that was I want us to be able to move like that properly to everything But that was still good because that Before that we had like what ten years of speaking about is also I'm pretty proud that it actually happened Yeah, but that was also like the most painful porting projects So When the DNF five discussion in fesco started we made an effort to Well try to build a list of requirements and this list was I mean there was some list creator was very incomplete And I think it's kind of the same Problem that appears in this kind of switches and I I think it's I mean one lesson for me would be that we need to build this kind of list and Building of this list is of such a list is hard because you need good insight into Not just the thing that is changing but all the surrounding tools and processes and that's hard And I think it's particularly hard in the case of of Of we are more DNF because there's just so many so many points which need to be touched and So many of the external tools are like Well dead or semi-dead somebody wrote them five years ago and they live in in length scripts or somewhere else and as long as they work Nobody will start them again. So nobody knows and there's no clear I mean Like I know we have federal review and Obsolution scripts and this and that and I mean in half of those things have no clear owner or no ongoing maintenance and nobody Will think about fixing them until we switch and then we realize that they're not working Yeah, so if we could do this again, we should probably try to figure out how to how to Find about those things earlier I want to add to that. Yes. I agree, but on the general topic of requirements. This comes up from time to time and Fesco to me Fesco should not be the body that is dictating Hard requirements for projects like that. I mean really when it comes down to to DNF like we defer to the change proposal owners the the engineers who are working on that but Generally speaking our job is to to vet these changes and say, okay Well, the requirement from our point of view is don't break things for users, you know Like so have you done that we need we need information to determine if you've you've really set things up that way And I think in this case the DNF case They they're needed there were such a broad impact that that team needed help to to really understand Who all needed to be a requirement so in that respect it was a little unusual But you know really changes or changes and you know that might mean that requirements change for software So we shouldn't say you know in a change proposal this team wants to do something We we shouldn't go in and say well. No don't change that because we like the old way, you know I mean that's just not really what we're supposed to do Another thing that actually that change proposals really do and I think do well is I agree with you that the team needs to kind of figure out what they're doing and propose something But like in this case DNF is so pervasive across the entire project that they Probably very likely do not know all the things so once the change proposal is out there people can come and say well You know my thing uses this and have you talked to me? Well, we didn't even know you were using it So I think that the visibility has helped a lot there and this is where I say this is the change process working as designed Because if you think about what a change is actual proposals about it's a description of An implementation of something that is going to affect the distribution and the community right so the community Responds so that provides feedback in kind and we as fesco collate that feedback and turn it into something In bite-sized chunks that they can that they can turn around and and provide something actionable like we go through and we Understand and like when it goes to a meeting we say like these are the things that the community Highlighted to us that are important, you know, please. Can you can you find some way to address these? so for making this acceptable and that's essentially What our job is at least the way that's how I've yet and we're doing and and the dean of change is and Actually both changes. We've talked about today how our excellent Examples of how we Arbitrate those changes and make sure that you know, we try to have the community's best interests At heart when we're when we try to make when we implement things in fedora Linux I have command and I concur with you and disagree with David I don't feel but fesco is just about vetting stuff, but like foster and then a Lot of people things that fesco has executive power and we Know that you have no executive power You can do anything. I would love to know where this executive power comes from. Yeah, but I think you have a huge power And that's the poking And we actually seen that when this be shake start like what's the status of the DNF 5 and that was the moment when this starts Happening to move and that was just just a poking so so like poking about the other stuff like pushing Our job is cat herders, so I think I actually we don't even have poking power We have mostly stopping power, right? I mean fesco unfortunately can say no, but it's very hard to Force any kind of action. Maybe as individual members we can poke stuff But as a body we can only say not this cycle Yeah, but they could also just ignore us and do it anyway. Yeah, they can ignore you but but but We mentioned here like we are waiting till the DNF 5 will become default And then the semi-dead projects will only fix the stuff and I think It can be Benefit your body group to like poke before like yeah, it's going to happen Please poke poke like change it change it, please. Yeah, you can if they if they will ignore Yeah, that's just a life and it can change from semi-dead to debt project and Well, I actually I think that's where change proposals can be proposed with Like more structure around them like like have have deadlines within it Like you know we intend to land this change initially give us some data points that we can do that poking by But I mean really Stopping power is kind of the the only thing we have and that that's why we have the contingency plan Requirement and too many change proposals, you know, we'll come in and say not applicable You know self-contained change or something and it's like well, maybe that's true But we still need a contingency plan, you know, we need the ability to back this out What do we do when everything breaks, you know, basically yeah Like I've seen a few change proposals that had been before they even reached the mailing list like where You know Ben when he was you know doing the thing is like yeah He would kick back ones that like had the section completely empty You would be surprised how many of them don't have one at all Before they get to the point that they get posted, you know, that's actually We maybe at another event we should do a presentation or a Workshop hackfest kind of thing on how to write a good change proposal. Yeah, so We've all written some yeah, and I I help people when when they have questions on how to put it together and it's it appears trivial, but It you know, there's there's key information you need to think about and put in there Although the hardest part of it is actually doing the work Well depending on the change proposal, but yeah, yeah, but see you can you can write the change proposal And then have someone else do the work. That's great. If you could get it to happen. I Mean one change proposal was literally one change proposal I did was literally in including one extra package Into a cons group. So it was a one-line change But the change proposal document itself was like I think six seven eight pages long and it's just Yeah, so that also reminds me of it was one that I think was a banded some years ago that was proposing There were some spec file macros the really complicated Sort of all the forge macros Not it was it was something related to that Yeah, it was just the change proposal was just so long and so detailed But whoever wrote it sort of abandoned it. Yeah, and and I don't I'm glad that happened because if we had to discuss it and vote on it I would have had like a crazy long meeting one day and and then never voted But yeah, actually if you look at the wiki for change page incomplete, which is the initial status There's a whole bunch of interesting things out there, you know You look at a proposal and you're like, huh, that's actually kind of interesting But it was somebody who started doing it and you know for whatever reason didn't have the time to complete it Or whatever and that's actually interesting to look through back through the history there And sometimes it's it right and sometimes you look at one of those things. You know, wow That's a really good idea. Let's see if we can revive that Yeah, I mean a couple of the changes I did were actually Me reading through the old features list when they remember when they were all called features way back in the day There's some old abandoned ones there and it's like oh this this is interesting Let's see if we can like pick it back up now that things have changed enough to make it better Like it's also if you if you're thinking of doing something cool or interesting or you had you want to do something cool And interesting and you don't know what you might want to do you could go look at the old graveyard of changes and And see if there's something that catches your eye and maybe you want to spruce it up and try it Just I'm gonna ask. Please don't revive the fedora k free BSD No, don't do the idea that that came up at some point. I don't know if it still exists on the wiki, but yeah someone There was a joke proposal several years ago. We hope it was a joke proposal I hope it's a joke proposal to to to add the k free BSD and then like add the Resulting infrastructure to be able to dual test everything on Linux and BSD. It's like no Somebody did that somebody actually did do that as a derivative distribution. I don't ever want to see it again Justin wait, wait, I can't hear you Sorry, I'll start building that currently coper we can we can get started now Yeah, and and if you do that can you also build the Darwin kernel too? Oh, yeah, I know there's a lot of people that would love to Darwin. Yeah Fedora Darwin, I don't like having hardware working. So, you know, that seems like an ideal kernel I actually IBM did some experimental kernels, too We should maybe throw in I believe they use Linux drivers to make it easier like the K13 kernel and things like that Oh, that's perfect. Yeah, yeah, then we're good to go. Let's go. Yeah So joke or not we needed on the recording and also nothing ever gets deleted from the wiki. So I'm sure it's still there So We had a question in the chat what all things Fesco is thinking to do to explain how change proposals work and how Fesco works to the larger crowd And we have been touching on this in some aspects, but maybe so There were a few things that actually that Matthew came up with that I think are pretty good, especially for the discussion side There's actually a video out there that describes the change process and being having a link to that for people who like video Explanations of things, you know, right there in the top also some more Explanatory stuff, you know in the change proposal itself Like maybe going to the process or a short summary of what the process is or something like that I think could help but yeah, it's It's difficult to to provide that context with somebody who's just coming into new to the project It doesn't know what's going on and we've added the disclaimer and stuff like that to the top But it's it's difficult to convey like the whole process But yeah, do we have a section in the documentation about the like writing a change proposal? You know, I think so yes. Yeah, we do we could probably expand on that Maybe do some blog posts just raise awareness as well. I think it's a topic that we should probably Be continually Revisiting just because people come and go like it's one thing to do a one-off, you know, and and that's good But the audience changes. So yeah, the video is a good idea and you know just increasing the Marketing and communication the heart the hard things to do The other thing that's really interesting and I thought of this earlier And then it kind of faded from my mind and now we're talking about this again So it came back is the way people talk about as if there's a fedora team that does stuff I would love to know what this fedora team is because I've never seen them. Aren't we all a team here? I guess I don't know would like but the it's really important to know the really important thing to convey is that a Change proposal can be done by anyone who is involved in in the fedora project at all, right? And there is no like team of people that can make them and do the things and whatever It's just any member of the community who is interested enough to drive an Improvement to the project. I mean we've we've seen the changes process be abused for things other than the actually changing things in the distribution so clearly it has gone a little bit further than that but Like it's intended for technical changes, but we've used it for project changes. We've used it for culture changes We've used it for a lot of things Anybody who's interested in driving change in the project It is the method that the whole community is able to synchronize on and understand what's going on and Get consensus on so I think it's one of those things that It's hard to do from the inside because we have been living this process for so long that we don't see how it looks for people from the outside and maybe some newcomer should take on this Job and and figure out how to explain it to people who haven't been familiar with it for a long time Well, we all actually stopped talking That happens every once in a while Any anyone have questions? More any in the chat? Nope. How are we doing in time? We can Okay Okay, so some fling so you would have executive power. Oh, no, what do you would change in Fedora? Whoa So I have I have one thing that I was I was recently looking at fail to build Tickets in my packages and other packages And I there were a few they were like there was a package that was preventing 50 other packages from building and I start looking at it and I see oh Actually, somebody filed a poor request to fix this package a year ago and it's been hanging there in This gift ever since and I know if we could have a way to to just have poor requests out to apply after a week if nobody Protests then that would like to see that Are you sure you want that because like that's a be careful what you add what you wish for kind of kind of a of a wish and also, I mean it's so so we have the package a dashboard which makes it easy to see all the poor requests but It's also Apparently very very easy to miss poor requests if you I mean you get a notification But I also get there may be a hundred other notifications every day. So if it flies by that's easy to miss and maybe there should be some Mechanism to escalate this. That's one thing. I Have one like it's it's on a lighter notes. I if if we add executive power. I Would like to bring back the fedora release naming and Yes, the reason I say that is that And I'm not saying it has to be implemented the same way But what I liked about that is it was a participation point for everyone in the community Regardless of their level of contribution or their technical skill Everyone could vote on a name and then we had like each time we made a release It felt like we were this community around that release and we would have that branding now We did make it extremely difficult the process and stuff like that sure I get that maybe we could fix it but I liked the I liked the release naming thing that we would do it's always in the back of my mind is that I wish we kept the code name stuff because It was it was a fun point like one of the earliest entry points for me was you know helping pick The name for I think was one of the names that was on the voting roll for fedora fedora 12 It was a long long time ago and like I I Think that that is like literally one of the easiest entry points that someone could have to have Some ownership of what we what we make as a project.  Yeah, you know wouldn't that be easier with the discurses Where you can vote just with plus one and I don't think it was the voting that was it was the names that were on the role The problem was of course as it often gets down to lawyers. So yeah, um it well the spot can Share some some backstory with that But it doesn't it doesn't look like you you really want to right now, but oh Yeah, so so maybe I don't know maybe we can get out of that business now, but I Mean just just from a community participation standpoint. I think it was really useful That said Of the release names that we did have What do you guys think was the the best name? Oh? Ever I'm gonna I'm gonna say Zod Shoulding there's a cat was shorter. Here's cat is mine as well. I mean we also tested the UTF supporting the distribution. So Yeah, I'm gonna go with that. Yeah Although werewolf was a close second because of all the kinds of combinatorial like the community artwork around it was phenomenal That's true. I I do like the Schrodinger's cat one because it did create the release blocker So on an apostrophe. Yeah, so the the act of participating in choosing the name actually led to Actual technical bug find, you know, which was which was neat And well Sometimes have the feeling that it's too hard to do things in federal like that Things that should happen and in general people agree that they should happen but they don't happen because stuff is slow and so sorry, I mean, I There's many places where I think we should as a community keep in mind that Just delaying things indefinitely is not a good Outcome and so it's so if I had executive power, I would just use this power to generally makes makings happen if possible I'm gonna go with that be careful be careful what you wish for for that one again because That can go very very badly. I Have another one on on on Exactive power. I'm gonna assume that we have an infinite budget. Oh, yes Let's go with infinite budget on the condition. So I would like to hold a flock and fund everyone in fedora to come to it Yeah, regardless of distance and all that stuff. So that I with the infinite budget now with the leftover money What would you guys do? with the leftover money of an infinite budget Next-generation koji with being able to do reverse step tracking all rebuilding because then Then I don't need to hear of proven proven package or request simply because they were the unfortunate souls of a library But they can't actually upgrade anymore because they can't fix everything I would like that kind of grunt work to just go away. Yeah, I was gonna say I could spend an infinite budget on Infrastructure, there are so many things if I could hire 500 people to redo this thing in a sane way and Fix the tooling make make things so easy for package It would it would be amazing. Yeah What do you see is the future of disk it in light of things like Packers Current maintenance status and people moving things to get lab and things like packet leading more people to Not really treat this kid as the canonical source for spec files anymore. All right Let's leave time for it. All right. I will keep I will keep my statements on this short. I promise Having pulling away from the canonicity of disk it is a recipe for disaster I have been in distributions where so some of you may or may not be aware that I actually am involved in much more than Fedora Over the years have been involved in over a dozen different Linux distributions Who have done a variety of different ways of shipping software? One of them, you know from a lot of those experiences one of the things I personally feel is a colossal mistake is Orphan in your agency of the method in which you ship software And so one of the things that we have to be really careful about With stuff like packet and what we are calling source get others called merge source trees or Or get package trees or whatever you want to call it, right is We cannot orphan our agency of What we ship because we are still ultimately responsible for it the distribution get or disk it is Our packaging get reposed that maintain our agency for that the canonicity of that hosting it inside Central Fedora infrastructure is how we preserve the integrity of our systems Projects like Debian which do not have that requirement Have to work around that problem by doing things like their reproducible builds efforts which Give say that for packages that are going into Debian We need to try to find a way that you can do a Reproduction of it from the output artifacts to recreate the output are in from the inputs produced by the output artifacts To create the same outputs again We have much more sophisticated ways of being able to Verify builds Then they do because of these other guarantees that we have that to be frank We don't talk about we don't talk about like why we've made a lot of those early decisions and what what benefits they offer The Packer versus whatever thing the maintenance status of Packer is In flux because we've got new contributors coming in who are actively working on the project earlier like late last year as one of the I was able to with the Fedora infrastructure folks finally get the CI Infrastructure working again, which means I could start accepting contributions and that has led to a Slow revitalization of the project and in a way that we're actually heading towards a new major version Where we're doing a lot right now. We're doing a lot of cleanup killing out a lot of cruft code There was a lot of weird experiments that we've done that are not needed. So We'll see how that goes. We have three minutes left or two. No two minutes left Well, I will be actually very brief also I Think we need to have another discussion about this. I know people maybe don't want to hear that Oh, no another discussion, but I think things have changed and I think this shouldn't be a top-down dictate You know, we shouldn't say oh, we're moving to get lab You know, that's that's what we're doing. It should be all right Here's our options. We could stay with Peg error. We could revitalize it. We could use it We could go to something like Faustodon or Gidea or you know those sort of products now That are out there and have the possibility of federated repose we could We could do get lab We could do get lab in different ways. We could run our own get lab We could let them run it to get lab for us, etc. Etc. All these things are a choice that our community should You know have in front of it and look at all the facts and figure out and maybe it goes to a decision from pesco Or something like that or maybe it there's constraints around, you know costs or whatever that that way into it also But I think we probably need to have that discussion again I don't think that it you know, we're going to get lab is a Conclusion from several years ago that we should stick with said that's my take Yeah, I I just want to add one one thing because you mentioned packet as well as where disc it lives So I separate sort of thinking about the workflow of package maintenance and and where source lives and and that sort of stuff Versus the the system you choose to put it on like that that is a there related problems with different what I would like to see is a discussion and decision in fedora about the Direction that we do so packet like if a project is going to adopt packet Building from upstream then I don't want to also see work happening in Disk it like I don't want to see patches and PRs there and have it be Bidirectional because that's really not like we should we should define what the flow looks like from upstream to downstream integration in fedora and I think I think that's important because it gets confusing with some projects So I think it needs to be handled the other way like I would all say that this kit is the canonical place And you are welcome to use any workflow you you want to put things there And I think packet is a very an option that this certainly growing in its usefulness But you it must take into account the The proven package of workflow and poor requests from the community. They just must be integrated Into I mean like any changes that you do must Take into account that things happening this get independently keep in mind that if you like let's say we go with the idea of like if you're using packet Then you don't have you don't people don't do into this kit that means for example proven packages are completely cut out That also means that other community contributors if they are not accepted members of the upstream project that's using packet They can't get changes in that also means if we do modernizations or infrastructure changes They can't do anything either like it's actually like an extremely painful Problem if we block people from being able to work in this case. So I'm not advocating blocking I'm saying that Fedora needs to define what those workflows are Because right now that's too ambiguous and and anything can happen and what I see is stuff gets lost Well, it's actually not that ambiguous the packaging guidelines pretty clearly say that this get is the canonical source for Spec files and patches and whatever but some people Think of all these reasons why they don't have to follow that rule or whatever it is Yeah, that and and that kind of goes to Communication and marketing and making sure everyone understands and is on board with with what what that says Speaking with my packet head on I get the problem is more like like not As Neil mentioned like not allowing people who has the right to do something in Fedora this get not allowing them the upstream but like when people use packet or other tools and maintain the spec file somewhere else and The this is just technical way where they put the stuff. So Koji has something to build and if people submit the Patches or poor request there. It may hang there for weeks slash months slash years We don't need to make that for it for us to happen So so so so like it's more about like letting people know that the majors actually do the stuff somewhere else and if they Want to submit some contribution. They should probably submit it somewhere else and it may speed up things So for me the canonical point of view like I Don't know what is canonical Think in in this context. So I think that this was a major design problem in packet. I Mean it was I Know it's it's sorry, but things will not work well this way and we'll dig a hole I mean like We have what 8,000 projects and with 8,000 upstream so we'll dig 8,000 holes for ourselves to to fall into Hey, I don't know that it's necessarily a hole. I mean I understand the problem, but like we're doing example Anaconda They I don't know if they use packet But they they do releases upstream and then they maintain the spec file there and just sink it to disk it but there are occasions where a compose will fail and Adam will figure out what the problem is and he'll talk to the anaconda folks He'll be like here's this problem, you know file bug a file pull request But I'm fixing it and just get right now and building a new anaconda so we can have a compose And that's fine because when the next thing comes along and it overwrites There's already a pull request because he knows that you know in the next version that they release He wants to have that fix already there. So I think it's to your point. It's more of a Expectation thing if you don't know that and you come along and you're like, oh, I'm gonna file a porcus or the kernel as Justin Will no doubt point out. It's the same way It's you know, if you're if you're filing a pull request there or changing something there, that's great But you should realize where you need to make that that's what I'm talking about. Is that that? So we do actually have well So we had documentation on the page when you went to the page saying with a link to come on our saying This is where it's maintained But we still allowed pull request I did actually turn off pull request because whether there was being ignored But I don't think that getting rid of disk it as much as I know a lot of people in red hat would say source It should be the canonical source. This is where we're working But I think this gets serves a good purpose there in that I do yeah, I overwrite this get now I check the diff every time so if somebody's done something I'll at least see it and Hopefully they've they've put in a merge request and and the source get but Having just get there. It does mean somebody can go fix it without waiting for everything else You know yeah, yeah exactly and and so that does serve that purpose But you also what we've just all said the expectation that you know, you're gonna fix it there But I know it's an upstream project that's gonna feed back in so I I understand how that works And I'm not gonna have that change loss, but right in the interim I need to fix it now. So the other problem is that we have not defined an expectation of upstream projects that wind up doing this because Here's the deal when you're integrating into Fedora and you're doing this stuff Ultimately the Fedora contributor has to win in a conversation Where something needs to be done to fix something and if that is not happening if there's no responsiveness in the upstream project There's no responsiveness downstream in the Fedora disk it something has to win something has to give and right now what happens is if somebody has gone to go Packety source kitty, whatever and and someone so contributes to submit to change upstream like they're doing it and No one responds this is a even worse situation than we have now because There is no agency. We have lost a way to fit to to realize a solution because we have deferred our agency to someone else who doesn't have to Care about what we're doing and that is the crux of the problem.  No, I'm saying that if you're following that workflow you consider disk it as a last resort It has to exist as one. Yeah, it has exist But but yeah, it's I think we are slightly heading to situation where we have to clear clearly say what what is canonical Source and differed from the last resort or some something to do about it and I probably With packaging we will probably get off in near future And we will see where we'll Had it us the important thing is that if somebody is going to use upstream project packaging in Fedora and Then ship it into Fedora They also need to understand that there's an expectation that they've got to respond to Fedora needs or otherwise We've going to have to do something right well I think the point there is that things can change Yeah, like say you have a project and you're you're maintaining it upstream and you're just putting that into Fedora Through disk it and everything's great and Rosie and whatever and then maybe somebody takes over the other the upstream project But you're still maintaining the downstream thing and they make some change And they're not responsive at that point you may want to break that Relationship and you may want to say okay now we're just going to maintain this here Here is the canonical source because the upstream is doing things that I don't understand or be not being responsive So I think there needs to be a way to tell people what the canonical source is and It can change it could be that it's not the same tomorrow that it was today Yeah, I think it's a big problem if we're saying that this kid isn't the canonical source anymore I mean one from the agency aspect and for the ability for other people to Collaborate on the distribution together, but also for the contributor experience like if one package is using get lab And one package is using code for another one's using one is using get that free desktop that whatever It just creates a very bad experience for contributors and we kind of lose that collaborative aspect And that anyone can go in and leave a review And I don't think that we should be you know trying to communicate to people that the canonical source is somewhere else I feel very strongly that the canonical source should be this to get Yeah, so this is you know when I look at the contribution experiences between Debbie and and Fedora This is the core difference because Debian doesn't have a canonical source requirement Which means that we do they do not have a singular path to contribution for any package It could be anyway Actually, you're not even required of a VCS for Debian packages You can just upload things and they will do that and then you're just gonna have to figure out how if you need to do Something how how it's gonna get done and that's not something I want to have in fedora at all on the other hand Who has more packages Debian or Fedora? We're How many of them are alive? Is it a Debian because and I think like yeah, it's make hard thing contributing to other packages But on the other hand like this situation make easier other contributors Taking care about new packages because it's more easier for them. So so so Very very good problems I think number of packages is less meaningful than it perhaps used to be because now we have rust and go and yeah A lot of that stuff is just sort of auto-generated, but you know, yeah Also, like you have to think about the quality and the usefulness and the liveliness of those packages There are a lot of packages in Debian that do not have a functioning Upstream or development or whatever they're just carried forward forever And they don't fail out of the distribution because of things like oh, they don't compile because They don't they don't do those kinds of things like we do where we fail out packages For compiling for not compiling on a year after year right like those are the way we handle our distribution Tilts towards active maintenance, which means we will always have less packages Than some other distributions that don't have that preference Yeah, and I personally would rather have more high quality packages than less low quality packages because Fedora is you know a collection of high quality packages that has guidelines and standards Not just a kitchen sink of every single possible open-source project in existence so We are 11 minutes past the I'm initially planned end of life of this session. So I think that I Mean this is a topic we could continue on for for an hour another hour. So let's just wrap it up now and Thank you all for coming. Yeah. Thank you guys. Yeah. Thank you. I think we can we can get started Yeah, hello folks My name is Neil Gump. I was here on the stage It's a little bit ago, but I have a new companion this time David Duncan. Come on introduce yourself Hey everybody, I'm David Duncan. I'm a well in the floor team I work on the cloud edition and I've spent a lot of time Working on that in my professional life. I'm a partner solutions architect and I work on cloud Cloud images and cloud solutions for Amazon Yeah, and I actually also wind up occasionally working with him professionally too because I work at Red Hat as a Senior black belt on managed OpenShift solutions and that means that I get to interact with him when working on Red Hat OpenShift service on AWS. So or Rosa as everyone loves to call it So we're here kind of to talk about, you know You know with you all of you. This is a participatory panel thing about You know cloud images and getting you know people to be able to make their own what kind of process type stuff and You know what we hope to achieve and what you would like to see From Fedora cloud to help make your lives better running Fedora in the cloud. I Think I think one of the things to you that we're we're curious about is Where in your position, you know, where in your workflow might you have a you know a good space for something like a bespoke image We spend a lot of time looking at what we can do to create images that are small minimalized and and provide Sort of a generic base, but That's not quite I mean, that's that's just the tip of the iceberg. You know when it comes to Come comes to how the cloud Cloud experience works, you know, you want to be able to Generate manifest and create Application space that is specific to your workloads, I mean there are many things that we we We do in our daily experience that That we don't want to do twice Effectively and so just to give you a great example the the core OS images that are created for building out the the OpenShift services for all of the public cloud providers is Is in fact Not just core OS it is core OS and series of tools and technologies that are necessary for deployment So that we can get that you know get that deployment model down or or decrease the amount of time to To ready to to a ready state Right, and you know you know like with things like reddit OpenShift service on AWS or if you're running, you know Maybe as we were talking about yesterday in our talk about Fedora cloud KDE, right like these are Scaffolds of workloads that we build to provide something useful and differentiated on On the cloud that that you will be enriched with and we want to know like we what we want to do is You know we want to help you be successful in the cloud doing these kinds of things and doing cool stuff with it and We want to know what you guys are doing in the cloud with the stuff to help us or to help them Yeah, just just as important is is what documentation do you think is is? significant to make the next generation of your of your Experience with the cloud edition better, right it doesn't have to obviously doesn't have to run on a public cloud Or it doesn't have to run on some managed cloud it can run on your desktop and and We expect that the experience to be just as important just as just as critical For for others and with some of the things that you can do now with cloud in it and in vert install and yeah There's a lot of ways to like there Like there are some of the things you may not be aware that the the cloud working Fedora cloud working group actually Maintains is you know the vagrant images are ours, right? Like we we take we take care of the vagrant work stuff I think we recently started doing what cube vert images or something like that and We're also doing things around Adding it to additional cloud providers we just added Azure recently and We're looking at Oracle cloud and a couple of other Providers just to kind of round out the table of things and we've always had the network of You know the common VPS is like your linodes and digital oceans and things like that that we've also been Having our offering on for many years So anyone got something that you want to? Yeah Hey, how are you doing? And my name is Brian and I work quite a bit with upstream Qvert So I actually came here today to ask you about if the clouds it would have an interest in Publishing images for Qvert VMs basically. Yeah, yeah, I think we have the definitions for making them But we don't know where to put them Yeah, so I think at the moment we're building our own kind of container disk images Which are basically just kind of wrapping we say for our cloud sent us cloud Images, so they rely on cloud in it and then we store them in quay as container images sure So Yeah, it's great to hear that you're interested in actually developing some of these images because that's fantastic Like our goal here is to provide all The deliverables that are needed for people to have an end-to-end positive experience using Fedora technologies in In the cloud development and production process So whether that's on your computer or actually in your infrastructure or somewhere in between like we want to we want to make that experience good and Help help you succeed in that front Yeah, so what you know one of the things that made us excited to talk about this today was that we wanted to talk about what tools are available, right and so a A couple of the things that are available right now is is Is just being able to create the build definitions inside of Koji and then build those in Koji And we were worried that you know, there are a lot of people who don't know or in our community, right? who don't know that they in fact have that up that that Ability ability. Yeah, just today right to generate those images in a way that is consistent with their expectations I know Adam knows about it because he does it. Yeah a lot We appreciate you Adam yeah and well and but he was also very vocal about learning and you know learning how to do it and and There's probably if you were to if you were to just wander back in in the infra lists You'd probably find some great great conversation around around that Yeah from him But we want you know, we wanted to make sure that everyone knows that these the tools are there the There are some other you know, there are tools for uploading to Cloud providers and and whatnot that are there and we while we don't expect that we'll be doing that for everyone Individually we expect that you know if there's an image that we need to have in a place that makes it available for Specific cloud providers that that's something that we would manage and maintain for you And then the you know, obviously OS build is out there to help you do that in your own independent Like in your own individual accounts. What?  We also I mean the tools that we're using in fedora cloud day You know, we're in the middle of a transition moving away from some of our Legacy tools that are kind of specific that can only kind of run effectively in fedora infrastructure To stuff that people can run anywhere. So whether it's OS build or In some images will be look we're looking at using Kiwi for and the idea is that we want to move To a model where everything that we're making is something you can make too Because like we know that as much as we want to serve and offer all these wide variety of things The key power of the cloud is being able to build for yourself tailoring it to your experience And we want to make that a core tenant of what we are doing We want everything that we're making to be something you can take and make your own Yeah, there's a lot of there's a lot of times where we get feedback from the cloud providers about you know security or Identify some some sort of vulnerability that they don't want to see in the machine image, you know as such They want to see it. They want to see a new image deployed and then that you know that that image has The governance around that is really is really complicated because we really want to just put out one image and then put out an updates You know associated with it But their process doesn't fit that You know fast forward to the customer experience, right? Then you start they start to have a deeper impact on their own customers their own customers think that oh, okay Well now we know that our process is to update our machine images and then and then or or just images depending on who it is and then and then The updates themselves are there for us to be able to make something that is more consistent, but then to make it like You know once once every two weeks or so, you know So we know that the people who are people who are working on Cloud solutions are also being you know have a high amount of pressure to issue New images instead of just creating a patch management model for for the instances that they create from those, right? Yeah, because if you're in the more ephemeral world and you have the ability to subscribe to a feed to get the latest image every time you You provision then you probably want to take advantage of that But on the flip side if you're one that's starting from this and you have a machine environment that's Sticking around for some time and you're going to keep it and lifecycle it then you you would favor You know being able to do patch management and things like that So being able to handle both types of workflows is something that we try very hard to also do but and that kind of comes back to you know, we were trying to make our tools easier and simpler and More flexible to be able to handle these divergent use models because they're they're both valid And we want to make sure that people are happy using these things in the cloud locally wherever Yeah, yeah, the Kiwi decision was one that that came about because we were looking at how to create Composable image definitions and it was just a fantastic way to do it we could create any basically any kind of image we wanted from the WSL all the way out and So where we were We've been pushing in that direction On the other hand, you know, the uploads and configuration and like post post config We started off trying to determine how we could build our own tools, but then it looked like Ansible actually does most of what we needed to do so So we turned our we turned the the cloud team has kind of turned towards Maintaining those The software development kits in you know in fedora so that we can have full support for the For the the Ansible collections that support them and ideally like these are things people can take and use in their own environments one of the you know, I spent eight years as a DevOps and doing DevOps type stuff and You know, one of the things that made my job challenging was that I saw all these interesting things and ways that people were doing stuff and Things that I could have loved to use in my own environment But they didn't make it accessible or available for me to be able to easily take it Adapt it and extend it for my own use cases. Well, it's still not easy to like Integrate with the infrastructure. Yeah, I know but I'm speaking aspirationally. Yeah So like this is something that we like David and I have been like working really hard on trying to like Make that a guiding a guiding point for us when we're when we're making our new decisions about how things are Supposed to work and what we're producing but it also allows us to put limits on what we are going to do and For example, we're probably not going to make a cloud variant of literally everything that that exists in fedora There we don't need to that but we want to but we definitely do want to make it, you know cloud variants that are that are going to do things that we think are Good a segue into more of a fedora experience So there are lots of people who come to those of one of the things that I talked about yesterday actually in the cloud cloud discussion was was the Was how we looked at the like cloud nine right cloud nine and inside of Amazon doesn't have a Fedora-based image, but it could and so, you know, it's for us That's that's something that we want. We we are looking at how we can produce It's I mean obviously obviously backlogged on on other things like getting rid of Python 2 7 but But but we're you know, it's it's a The you know, our our goal is to see how those things can fit into the process that you know processes that The users are already dealing with and say hey, I you know, I I couldn't use fedora as a as a foundation for this instead of just using You know some other distribution Yeah, and like it kind of riffing off of that a lot of this was well, you know Sys admins that are like in an emergency scenario. They're away from their main computer station You know being able to spin up like say a cloud Katie desktop that's got like, you know They're open-shift and Rosa command line utilities their cube definitions their access to their their version control systems All that push button provision so that they can do emergency work In the environment that they can actually be comfortable with I mean god I don't know if any of you have been assisted been in the mobile phone on-demand era like like I've been and But that sucks typing into a terminal and having to like do things from your from a from a crappy phone console That's in Android or iOS compared to like being able to access like a remote desktop from a web browser is Yeah, I would take the remote desktop in a web browser because then I'd have access to all the tools that I actually need to Function and and that's the kind of stuff that you know We're looking at to like enable cloud experiences, but also We want to have a framework where like let's say for example You know the Python classroom the Python say who maintains the plot Python classroom Lab for Fedora wants to also have a cloud variant They are perfectly capable of reusing our framework to be able to provide that on their own as part of their lab We don't have to do it for them. No, we would give them the tools and the capability just do it and make it part of their own deliverables we want to be able to provide that kind of flexibility to people in The same way so that they can actually have their own tools for success. Yeah, I mean I think we think other a lot of other groups are feeling that They they see that segue and we want to help them or not. Oh, sorry. Yeah, we want to we want to hear You know, we want we we think that there are other members of other teams that are looking at things that they want to be able to Stand up quickly. They want to verify You know, they may be working on windbind or they may be working on on, you know key cloak But they're but there's there's a lot of things out there that you want to just be able to to bring up look at and Drop and and that process of building those machine images and making sure that you have the updates and that you have sort of a, you know, basically a step You know kind of a step function style approach to ensuring that you have what you need We want to help make that make that a reality for lots lots of other groups Maybe you want to work on open QA on and on a virtual machine, right? Perhaps you want to be able to make your own just Environment to do all this stuff just with pushing a button in the cloud Any other questions? Yeah, just keep it So you mentioned that members can build their own images. Is there a good starting point for that? Is there a cloud sig docs page or is there a good starting point for that? I don't think we're quite there yet. We're trying to I mean, that's what we're doing That's part of our goals with the rework of our image build stuff In fact, that was a little bit of what I was hoping to to get us a springboard for today is to see where Where the interest is and how how we want to how we want to frame frame that documentation Yeah, so like part of the reason we did the cloud Katie you talked yesterday and why we're doing this panel today We wanted to have some interest and learn from y'all like what kind of stuff would you be? That you would find compelling and interesting to do With fedora cloud stuff and because we have some ideas of what we want to do And I and I think we're on a good path, but we also want to hear from y'all To see like what are you interested in? What are you looking for? What kind of questions? Do you have about like doing? Stuff in with fedora in the cloud and that sort of thing Yeah, I mean, I'll be honest the Kuvert thing is super exciting to me because You know, there's there's two aspects of that one Just generally Kuvert, you know, it's a great way to to create a great modernization strategy and to It segues into some of the work around lightweight virtual machine Environments that we have not gotten an opportunity to work on just yet But you know porting to firecracker and things like that is a big deal Also, congratulations, Kuvert for reaching 1.0.  Yeah, we finally got there. Yeah And you also mentioned Rosa on the site. I was just wondering is is container native virtualization included in Rosa? Um, I don't think so. Can we I don't yeah, I don't think so thing Yeah, we so it's it's a him. It's a him question for whether we'll ever do it though so so Whether or not it'll be supported. That's a that's kind of a question. Yeah questionable, but then but like There are flags in the metal instances where we're still working on the metal. Yeah, like most the reason why we can't do it is because The instance types we support literally don't let us and also it just To some level it feels weird, right? I mean, I think there are reasons where I think it would make sense but like you could also just easy to and And so like but I also can get the idea of like you want to have The abstraction and the sim the same API everywhere to be able to do all the things But we want to do your testing and yeah nested vert and then there's that and I mean like part of it So this is going to step back into peeling back a little curtains about fedora cloud stuff Part of the issue that makes fedora cloud images currently today so kind of painful is because the tools Currently require us to boot up into a virtual machine to produce the cloud images Which and the cloud and the virtual machine boot up process is kind of special and That makes it difficult for people to be able to replicate our processes on their own And I want to move away from processes that are so complicated That like someone who's like just getting started fresh into the stuff and is excited Doesn't get so discouraged that they want to flame right out of the whole thing I mean you can you know you can flatten a kickstart now and just figure out what's what's in the You know what's in the process, but that's like that's just one way to make a virtual machine I mean you could literally make it in the same way that you make an open VZ You know or made a note made an open VZ image. I hope you're not making open. Yeah In the same way that you know we did a gazillion years ago and that's that's not That's not unusual. I Know the Amazon Linux team when they when they build in it when they build an image It's basically just you know, it's a tar file And I don't think that You know You know, this is this is also one of those places where there's more than one way to skin skin You know skin this cat and and I think that that's that's a an important part of the Of what I think is interesting for us is that we don't necessarily have to do this in a way that is consistent with the expectations of The the one tool, you know, if we decide or determine that there's something that we want to do here That's different. We can we can we can be versatile and as long as it's maintainable Because I what we don't want to wind up is in a situation where we have a litany of things where we don't Actually know how we can keep them going Agreed, but you know if we're helping somebody sure work on their process I don't want I don't want it to be constrained to like oh, well, they're using kiwi. So no That's not that's not what I mean, right? Like I if a tool is is kind of central to Building the thing that they that they want or need and we need to and as long as it's Straightforward enough to integrate and and and support. I don't have problem with it Yeah Yeah, so there's a lot of ways there's a lot of different directions you can go with how you make your you know, how you create your or how your process Whether or not that process is consistent with the one that we have and I think I don't think that that's that should be something that would concern you know concern you The most I mean to me the most helpful thing is knowing that you can pull this you can you can pull together this Configuration you can push that into the utility the tools that we have already in place you can get your build alerts in the same way that you would get them with any other Fedora projects and You know you can throw it you can throw them over to Adam for testing Or Adam might just do it anyway and then tell you he did it. That's right any other questions any other Comments comments. Yeah, there we go This is more looking farther into the future But one of the things that I'd like to do eventually is more or working with like GPU instances in Fedora and that's Predicated on getting all of the stuff packaged in Fedora so that it works. Yep But is that something Assuming we can get everything to work is that something you think would be reasonable? Yes actually so like We briefly mentioned this yesterday during our cloud KDE talk like GPU accelerated instances and being able to support them properly in Fedora cloud is absolutely on a roadmap of requirements Most most of the most of the public cloud providers also have the ability to distribute the NVIDIA controller drivers so we can we can we can cheat piggyback off of that. Yeah I guess the the one of the problems I'm seeing is where we're starting is with AMD not with NVIDIA that's fine too. I know of one cloud provider who has one instance that has an AMD GPU in it And it's not current. It's a couple generations old. It's better than nothing, but yeah Like I was just saying I'm just all the cloud providers do have NVIDIA There's Amazon has one instance that has an AMD GPU in it And I have not been able to find a single other cloud provider that has an AMD Well then props to AWS for having AMD Radeon GPUs. It's pretty good Yeah newer ones would be awesome. Yeah more new ones because it's not technically supported by Rock-M There's questions. I mean it's not on AMD's official list Well, whether it works or not is a different question. It's just it's not on their current official These are the GPUs we support and then the one that's in AWS is different. Well, I suspect you know I'm not gonna speak for David, but at least from the Fedora cloud perspective I think if we start having those toolkits in place and being able to have instances Have images and in with that stuff preloaded that they that instance that when they're provisioned on those instances That they can just take advantage of them. I think they're gonna see a different dynamic start building up for it But you know, what do you want to talk about on the Amazon side? Well, yeah, I mean so from from the Amazon side we we have our own Crafted drivers for those so Excuse me the The way that works for them from the Amazon perspective is the drivers are produced by AMD for For the AWS environment and then they're maintained in an S3 location for for it for the instances But they're available and they're easy for us to distribute and there is a rail workstation That's that was created for specifically that reason to have the To have two things that incorporated into the instance one was the GPU drivers for the Jeep for for both Standard GPU workloads in general purpose GPU workloads and then to have on top of that the the nice DCV VDI solution in in place so the server is already there you just connect to it with a client and it's basically like running Running it where you are and the reason we wanted to do the KDE Workstation is because we don't have an upstream kind of of experience for people to have have that same You know to iterate on for this for some of the same reasons so We're really super interested in making sure that you know customers have our users have that available to them Because the user I'm the user experience around machine learning on Fedora is a story that we very much need to tell Yeah, I know I did it. That was part of one of my talks yesterday. I was You know what it is and hopefully how we're going to fix it. I am for for my perspective. I am super excited about seeing The ability to use GPU compute in Fedora out of the gate Seeing the Rockham stack it integrated in and being able to start doing that kind of thing I hope that it what it will do is encourage more GPU compute workloads, whether it's a IML or something else I mean render farms are another example of GPU compute workloads that Will use AMD GPUs with open drivers and an open Software stack that you know everyone it gets to exercise The flexibility and freedom that they need to be able to do what they want and not what somebody else prescribes of them Well, and just like you're doing you know Our goal is to build this narrative so that people don't forget or don't think that Just because there is another narrative, right that this one is is absent We want to make sure that this one is is very clear and you know in there Within you know within their space So for me, you know the machine learning. I mean if If you'll let me soapbox for just a minute The machine learning conversation is one that I think is really hard because first off, you know most of the applications that make this space Or work in this space are impossible to package, right? I mean we've had we've had that conversation several times and then The way that others have approached this Has been to take all of that Stuff that they can't really figure out how to package throw it into slash opt create an image and then just just make that available And that's how kind of you It's a scary, you know to me. That's not it's not a helpful model, right? Like there's no way to do updates. There's no There's a consistency there That maybe I I might like as a as a scientist But the but the real But this goes back to our like the whole idea of developing images for yourself We want to have those images in you know like in my project in my account, right? I want to have those those Those images but that doesn't necessarily mean that I want to find them out there in the world, you know In in an incredibly vulnerable state for you know for super long or I want to you know I want to find out that they are they disappear at some, you know, amazingly fast rate That makes it impossible for me to have a steady workflow so I mean I would love for us to to to To try and figure out how we can do that, you know Akash deep and his work on on the neuro neuro fedora. I've thought was a great a great entry way into this space and But it's but it's not, you know, I mean open data hub right was was a really important part of that process around the open shift experience and and you know, we we have I mean I have built out machine sets, you know that were that were GPU enabled for for doing machine learning workloads and in in the open shift space but Really a lot of that work can be done on a single machine in a very, you know I've sort of a limited space and so you can bring up one one instance Create, you know, do whatever it is that you're doing against what whatever data lake it is that you're accessing and then turn that off and Helping people to understand how to do that Means giving them the tools that they need to do the work Just I want to follow quick on something that you had said a while back to make sure I understood So it is is it is not a requirement for for Amazon to have the driver supplied by AMD It's just something that was enabled So if we had something in fedora that was completely upstream and didn't have AMD's binaries That is an option for yeah, that's only an option. You just wanted to make sure I understood correctly Yeah, you can use the open the open driver stack in there if it's pre-loaded Amazon just happens to have a bundle that you can just source it and install if you don't have anything, okay? Yeah, without getting without getting into trouble No, that's an important part Anybody else then I'll just ask I'll ask Isaac's question. No, we don't have the risk five images Just yet. Oh I You know I have expected someone gonna ask it like and I was so prepared He's leading the witness here. So I'm gonna I will I'm gonna ask two questions You just asked one of the questions, but the real question is when will Amazon? support risk five in Their cloud public cloud. I'll defer to people who are you know more more capable of answering that question That's the chicken and egg. So when Amazon declare that they're gonna do that then we can say okay, so when are we gonna have fedora? Images to sit on the risk five boxes. I Mean, why would we have to wait for Amazon to that because we fedora cloud does way more than that So fedora cloud is also, you know, we can you can run these images on Personal hypervisors you just boot them up with vert install and pass in the cloud the cloud in it user data or you use You know, you're using cockpit like in there and then that actually will boot fedora cloud images as as VMs Or you run an open stack on risk five or you're running Cube for like there's lots of different ways that this could happen and that's part of the fedora cloud thing There is but he's mr. Amazon. You're mr. Fedora, right? so That was a blunt question for Amazon and from a fedora Perspective then you still have the hardware enablement conundrum, right? Yeah, well, so yeah Yes, we could do all the things you said, but we have to have the hardware enable to begin with it's customer driven and For us it's somebody's got to give us the stuff to be able to start doing it because we're going to help triangulate the giving of the stuff to help you do it and We're also going to Politely encourage all the cloud vendors not just Amazon to support. I mean you can It's on public record Ali Baba actually have a tea head where they're developing risk five servers already And and their hook line and sync bought into this equation I would love to see Amazon do the same. I have nothing to say I mean, you can't you can't rule it out, but you can say the roadmap is definitely driven by based on What we're what we're being asked to do And all I'll say is that risk five is an interesting architecture, and I hope that it's bring up goes That the machines bring up go better than previous machine bring ups have Probably a small question, but do customers not want risk five on Amazon? I mean You're asking the wrong guy, but yeah I mean, there's you know, there's some other people who are more deeply aligned with the you know Peter DeSantis would be the one day and so to ask that question not not David Duncan and For speaking of people in the third person Neil Compa is just like well, I I would like to have As many architecture supported in Fedora and and offered in platforms and be usable in all the different ways But I suspect but I suspect that really it's just most people don't know that they exist right like you don't ask for something you don't know exists and and once you know it exists and You find a reason to want it then things the dynamics start changing And that's true even in what like the Fedora open source case Like we started having this interest and development into all these you know These architectures because people come in and they're like we really want this and they start talking about it And other people get excited about it. It's that flywheel of of success So I just like that to that that's yeah, it's a bit premature and early yet for Enterprise servers that are risk-based But they are coming and if anyone's familiar with the guru or the demigod or from Mount Olympus called David Patterson David Patterson maintains that risk life is actually going to surplant all CPU architectures over the next ten years And he's a you know, he's a former x86 guru etc and he's ridden the wave at arm and Risk 5 is something that's it just gives so much choice flexibility To the HPs of the world the startup chip vendors of the world's customers countries like India, China, etc That it's an inevitable Equation, but we're not quite there yet. So at some point, yes the Goldman Sachs and the Bloombergs are going to start consuming this and Amazon gonna is gonna spring a surprise on us and Hopefully we'll evolve on the groundwork to to enable all the hardware on donations then we can be off to the races. I mean That's very lofty my comment on that is that I remember when James Hamilton said that he thought that arm was the was the future and Then it was like three years before yeah Yeah but three, you know three years for him to make it make it work and it's very interesting to see from the you know from the perspective of of a An engineer Because the questions the question, you know the questions that That happened as a result of that were like More economic than they were than they were anything else so it's it's very interesting and then and then of course like a series of really amazing engineers started to solve really big problems really fast and You know, it's just like that seeing that the way that hardware can evolve in such a very short amount of time always makes me it always surprises me and and I Feel, you know fortunate to have worked with a lot of these people I Will kind of add to this a little bit You know your comment about how it'll be everywhere and surpass all the other architectures and whatever The the concern I have right now is that when you look at how risk 5 is evolving and how Development is going and manufacturer interest is growing and how CPUs and all this other stuff I Find some degree of credibility that there'll be more chips risk 5 than anything else by the end of the decade I Don't know if I could put a pin and say that it means that risk 5