done by you know building together a community of volunteers interested in doing this so we're getting better at this but we certainly had this problem over and over and over again um I love this wallpaper it's cool um uh one of the big things here Red Hat surprised everybody by buying CoroS the company they didn't know what to do with CoroS the Linux distribution um and so I fought pretty hard to have that fedora be a home for that and I'm glad that I did if you look in a different talk come to flock our next conference I'll show you some statistics Fedora CoroS is doing really well um and from now on at least through this talk um we're basically 180 days apart released release um very very consistently plus or minus one I think here um fedora silver blue came out around this time and based on the CoroS technology um which kind of showed that you don't have to just you don't have to be just a trivial change of packages um how do you make a face what have I done oh oh we didn't um yeah yeah whatever um we'll talk about that later um yeah uh the idea is basically you can make a big change to make in a spin it doesn't have to be just a different desktop technology and package in the same way you can change how the OS is put together in fundamental ways and also have a Fedora spin that shows that so I think that's a big fundamental thing okay um running out of time kind of move on we dropped the 32 bit x86 that's a big thing not enough people to maintain it um this is a cool release um I feel like that's like a really nice 80s throwback wallpaper there um yeah there's plenty of other stuff but um big highlight that Lenovo decided that they would ship Fedora Linux on you can buy it out of the box with that and one of the amazing things about this is first of all if we didn't go to Lenovo to make a deal they came to us because of customer demand and more than that uh they didn't want us to make a customized version they didn't want a customized version they wanted exactly what Fedora as a community was producing with no changes they wanted the real Fedora Linux not some sort of Lenovoized version they didn't want to take control over the kernel they wanted to deliver to their users what we are making so that was a really strong vote of confidence and a really nice way to work there um there's been a whole bunch of supply chain problems and COVID and whatever that and there's still not as not available as much as it should be but um you know a call Lenovo and ask for this and because that's really the sales people hearing it is what's causing it not to be available worldwide um COVID times we had Nest instead of flock virtual meeting everything I was actually kind of dreading that um thing but it actually turned out that our virtual conference turned out to be amazing and energizing and um I didn't organize it so I can say this um it was the best conference of all of COVID all of the Nest they were most of the things I went to um you know Red Hat Summit sorry Red Hat Summit people were just kind of draining and awful um uh it what and you know at the best they were like oh here's a webinar and our conference really felt like the community coming together from around the world and having a virtual party um and I that was amazing so um again for our community is wonderful um this you know is the peaceful universe wallpaper I think escaping the stress side that was a nice one there's a night version of that um here's a case where GNOME 40 um remember back to where we lost all the users this is a really big user interface changing things from horizontal to vertical or the other way around um and I think both Fedora and GNOME upstream learned a lot from this actually spent a lot of time doing actual user interviews and research before putting the change out there um and actually you know the results ended up being great and I was I was worried I thought there would be drama there um and it turned out to be there's some people aren't perfectly pleased you're never going to make everybody happy and I think the big change landed very smoothly um pipe wire a big audio change landed you know there's a little bit of rough there as well but this is actually a funny marketing lesson which is obvious in retrospect which is um a lot of people shouldn't ever care about their audio subsystem but there's one set of people who care a whole lot and that is youtubers and podcasters and suddenly around this release we were incredibly popular with youtubers and podcasters who had an exciting thing to talk about and they really liked that and so that kind of jumped up how people were talking about us so it's like oh yeah maybe appealing to the um audiences that talk a lot about what they're doing is a way to get people to talk a lot about things and that's it um and we also have our new logo here um Mo Duffy does a great talk about why we have a new logo and how the designing went into that um I really like it and I'm glad that it now um will not have fedoro as one of the top um typos um our searches from people who don't recognize languages where they don't recognize this as a word assume that that's what the old logo said because it had an A it was very indistinguishable from an O yeah um okay so um up at the time you know best release ever fedora linux 35 um this is the new overview the new um sideways um we didn't quite make the 182 days there so I guess more than plus or minus one but still reasonable range got things right instead of rushing um and here um some of the big changes that are not there we uh launched the chat dot fedora project dot org moving from IRC to matrix and working on discussion dot fedora project dot org I know some people are very resistant but I want to move as much mailing list stuff from there to that as well as a modern friendly interface that um is transparent to people who are not necessarily uh deeply involved in the project I also want to make sure that people are deeply involved have a good experience so we'll work on that transition it's not going to be dumped on people but I think that's actually a really important thing uh we also have a renewed fedora ambassadors program around this time that's kind of getting ramped up after covid um we're working on some investments in mentoring had a mentor summit I think that was an exciting thing um and this website new websites team that I talked about started coming together as well and that was really with kind of lessons learned how to make this a team success rather than something that could be it was just an individual I think that was really really good um uh yeah um right there uh Kino White was somewhere on this we started having some different variants using the OS tree um that were coming around was that was that in this release here Kino White was that um um yeah so um I was asked at the beginning where's the net missing three releases so uh I started adding on to here but I realized I didn't really have the perspective to talk about these most recent releases so um yeah I I'm sure there's interesting things to say a lot of stuff has happened in the last you know year or so um but I'm decided I'm going to put the seal on this talk here and end it and then maybe in you know 15 years someone else can do a sequel and talk about what's happened over the time there um but that's uh yeah I can stop this here uh so quickly um some of the harder lessons what what should we not repeat um you need to find a balance when you're making big changes um you can't just document it away you've got to make sure you're listening to users and you've got to um you got to make room for mistakes but you also have to listen and deliver what people want um but also having a community that cares about the project and is you know working together uh it can make the rough parts actually functional okay I'm running out of time for questions um yeah community teams you need to have momentum to serve to keep going long term got to make sure that you're watching the people who are in in positions where there can be a bottleneck so that they don't don't use one this is one of the official fedora colors so that's there there we go tune to that um yeah don't um yeah um we've got to make sure that our teams are have you know they're not just depending on one person that people should feel like they're recognized and awesome and supported but that also if they want to go do something else they can and the thing they're working on will continue and that should be a pride of success um and going kind of back to the thing that I talked about um red hat you know letting letting go and letting the fedora community lead um really has brought a lot of innovation into the project and I think that's one of the things that a lot of projects really could learn from it's scary to do it that way sometimes it's hard when you have to figure out how to get a community to come along with the decision that you want to happen but you get better results that way and okay there I have left some time for questions here so let's go to Peter Robinson is it about how I've gotten the architectures wrong the first what release was that in yeah Fedora IoT I think I mentioned it somewhere there but I may have just skipped over it Fedora IoT is an important addition that we added kind of one of these use cases that we thought oh we need to make sure we have something that covers this emerging area in technology so that's a very important thing actually yeah and actually going through the idea of adding a new addition made us come up with a how do we add a new addition process which we had not had before so we have that as well um yeah that was the thing Dan yeah right so there's another big change we switched to something called C groups v2 which is like a kernel level thing that's used for making containers be container-y I was almost going to say containers contain but then I remembered that I got Dan in this room so I shouldn't say that but yeah that was a big change that along with the system dn those things it's one of the situations where if you if no one takes that first step it's never going to actually get exposed to users and it will never get there and things will drag on forever so at some point you've got to make that change and so finding that balance of when to go to that and when to not is important and I think having in some ways like the additions split where I think was coro was slower in that some of the additions decided to do that yeah right so some of the some of the releases decided okay we won't make this big change right away and kind of rolled it out slowly so that people had the option it was the default in a lot of things so fallbacks where you okay this was is going to just work yeah so actually that joke why did I I skipped over landing my joke here I've done another version of this I landed the joke properly but I was I had spent too much time on other things here yeah so there's actually several other times it's mentioned we actually considered butterfs throughout the years and every time we did that we'd be two years off and then between red hat storage internally this is actually one of these let the community do things red hat storage people decided we're not going to wait for this anymore we've got to focus on what works and so they put all their effort into a thing called Stratus built on LVM and XFS and so on to kind of give the same kind of features there people in the community I did about things Butterfs could bring people coming working in with AdMeta where they have a bunch of big investment in Butterfs finally said okay look it's ready let's do it and so we went through our change process and with a lot of debate and with the red hat storage team weighing in to say please don't do this we decided you know what we've been considering this for years and years it kind of brings some neat things that we our users would like to see we're going to do it and this is again a thing where you know red hat doesn't call the shots and so the red hat for four things like that figure out how that Fedora can be a good upstream and still have different decisions on some fundamental engineering things but yeah we eventually did make a decision to do that another one that's kind of along the same lines is a change in baseline architecture which is basically as new CPUs come out they get new and newer and newer features and so if you compile your code with a newer CPU it can have better performance but it won't run on the older systems and so red hat for rel wants to update to the newer version of that but we have an ironic problem in Fedora where because we move fast we can't move as fast on some things because we leave people behind so with rel since it's got a long life cycle if people can't use the next release on their hardware or their old hardware that's fine they'll keep running the old release for you know as long as they want to pay for it being supported I think those like probably rel four out there somewhere being paid for by somebody red hat will take your money but Fedora we're kind of going on to the new thing and we can't say okay well sorry all your hardware from five years ago put it in the trash by new computers so we had to move more slowly and so one of the things I actually asked the people at red hat who were working on this were like well Fedora will never accept this and I asked them to bring it to the community anyways to discuss it which we did and it was a big discussion but out of that we kind of came up with this idea of having something called ELN which is a build of Fedora with defaults that are set for what would come the next rel so we actually have a place in Fedora where people can experiment with here is a way to do things in a different way so we can do those builds even if we don't make that the mainline thing that we deliver to people and you know in some ways you know red hat is special in order to do that because they're paying for a lot of the resources to do that but if anybody else wants to come along with a thing like that and can help provide resources so we can do it Fedora is open to that so if you've got some other big idea you want to do Intel if you would like to do your clear Linux thing as a Fedora build come to us we can make that happen yeah what is the next big technical change for the Fedora project predicting the future is hard I think it's both a technical change and something that is going to be a kind of a social change I think that this RPMOS tree approach that's used in Silver, Blue, Kinoite CoreOS IoT I think that's the way we want to go for all of our defaults at least it has a lot of advantages for delivering a system that's consistent you can do cool things like do a bisect to find exactly where a problem came from and it kind of goes towards a container focused separation of what is the base OS and what are your different applications to solve the problem we're trying to solve modularity which is that everybody has different opinions about the speed that they want different parts of their operating system to go and that approach lets you separate things in a way that lets you do that so I think that change towards that and whatever form it will take is probably the next really big technical change there's also the social change of convincing everybody that's the right way to do it yeah so the question is basically with the changes in CentOS project would Fedora make sense as a cloud server operating system in actual production and I can't necessarily speak to your particular use cases there are hopefully reasons for redhead pays my salary I hope that there's value that redhead's providing for that money but I also think that can be used in production and people are using it in production in some pretty large scale things depending on your ability to work with the community on things your tolerance for risk and your configuration and setup your change management processes yeah, you can definitely use Fedora in production for things like that we can talk later more about your specifics as well does that cover what you're asking or maybe it's crazy in a good way the right kind of crazy it can make it really work I guess I'm out of time here thank you everybody hey, hello good to have you here I'm Karolina I'm working as a software engineer for redhead but today I'm here not as a company representative but just myself as a person who's talking with open source not so far a time ago even though my experience in IT is bigger than that and inspiration for that was that I haven't met much other people like me who just started so I thought it would be valuable to share my experience and my observation how this whole process looked like and how did I start it I'm not very great about talking on myself so here is just a little bit of a subject and concepts I'm dealing on my daily basis job I mostly work and contribute for RDO projects we are shipping RPM packages of OpenStack for CentOS so mostly I'm doing packaging developing and debugging also some coding but let's go to the question how did I meet open source when I was starting IT I was fascinated about networking and security and it became obvious for me that you can do it without tooling which is available for a Linux and tooling for Linux mostly means that they are open source so I did two large students projects first was for my engineer project and this was a sandbox network for security research and testing and this was a kind of honeypot I was using technologies like Snort, like Nmap, Kali Linux and SecurityOnion and then on my master sizes I was testing and benchmarking distributed files systems a few time ago after I was graduated I realized that two of the systems I was testing Glaster and Cef was a Red Hat project if I didn't know when I was doing my job Cef appears best from all the that I tested then when I initially knew some Linux tooling I was really amazed that this wonderful tool is so powerful it's open, it's free for everyone and that idea was amazing for me that's why I meant to foster them this is a large conference for open source in a brothel and then I saw some community from a closer view and it was an amazing experience for me how I didn't met open source was on students' interest groups and it's a pity that there is no at least at my university any way to encourage people at the university when they are actually choosing their way or their careers path to encourage them to go to open source also, none of my friends or colleagues from studies was contributing to open source I was lonely in this topic why people enter the open source there are many reasons and they are mostly mixed but there are some few which are most solid to say generally people would like to gain a new skills they are starting new projects some new adventure to just learn new things and they decide to just share it with wider range of people also, they are starting something new because they need it for their work or for their other project and just put it on a github as an open source on a proper license sometimes they just need development in the desired project so they just missing some features or they discovered a bug which is really painful for them and they just decide to fix the job or did the feature and that's why they appear to be open source contributors and the third way I would say rare but it's going to be more often to be hired to contribute to open source project and this is my path of growing in open source and speaking about motivation let's say there are also some more soft non-technical ones like teaching others people like sharing the knowledge people have a need to belonging and all these motivations are mixed but it's worth to remember that most of their motivations are not materialistic no matter what motivation stays behind people somehow has to start the work and I observed that three mandatory pillars of good start boarding, mentoring and documentation are based off start if any kind of projects no matter if commercial or open source is an onboarding process for open source projects is crucial for gaining new contributors and for commercial one it's providing smooth and quicker start and also it helps to reduce stress and just make the start quicker so it's good onboarding process is shortening the time when people stops to be a cost but it also provides the value because don't get me wrong here even the best engineers are a cost at the beginning of course until the time they get into the project and get enough knowledge to provide value speaking about onboarding it's worth to mention some kind of vicious circle if they have a project where contributors or developers are overwhelmed with a work they don't have the time for people they don't have the time for onboarding or the stuff let me alone, I want to just work but if they don't onboard new people properly they still have too much work because they don't have anyone to help them that's why it's very important to have this process because it will help with it and automate some stuff for any new people everyone hates writing documentation including me but good documentation your support not only for newbies but also for experienced contributors and maintainers and this is a far more important tool than just the opportunity to say RTFM think about it in a more materialistic way when there is some process it's undocumented every new people who is dealing with it has to do some, let's say well engineering on it and it costs a time it's a cost so if we have some I heard a quite good description about such situation it costs tribal knowledge and this is a piece of undocumented project which is so obvious for a maintainer that they didn't even mention it in any kind of documentation it can be for example setting a flag before an infra-deployment and you know, you just start your deployment you are doing everything and you are meeting all these horrible bugs before that and you are struggling with it this poor newbie is spending a time digging inside and waste a time I know that trying is a way of process and a way to grow but come on, it's pointless this time can be spent for something more valuable and it's enough to have this flag documented and that's why it's called tribal knowledge also speaking of documentation there is one important note keep it up to date because if your documentation is outdated people who are using it users or new contributors are learning that they cannot trust this documentation and they will keep you asking anyway and this documentation is not needed because it's outdated and the third pillar of a good start is having a mentor as a mentor I understand the person on the partnership level will show you the project will explain you the rules show you how the team works but also this is somebody who will grow you in other ways than only technicals so this is somebody who also understands what do you like who will help you to clarify the way of development you want to go as a developer the good mentor should also understand the tasks and helps to provide the proper task which will grow the newbie which newbie will like and I have to say that I had an amazing mentor on RDO and if you will only have a possibility to find a mentor at your company please do it because I think that having a mentor is a kind of highway on a road in all this stuff I mentioned there is challenges first of them is a remote life and if you work for a corporation you at least have some chances for an office if you work like me so 100% remotely prepare also for a remote communication and kind of remote life I have to say that COVID has developed the culture of remote working in Poland where I work COVID shows for many employers that remote working is possible it's efficient and people actually doing the job when they are at home and also I have to say that it proved it to me because before COVID I never thought that I have enough courage self confidence to even try to work remotely not in a native language I have to say that COVID just leave no choice and speaking about language English is a not native language for most of us and this fact is often neglected people are afraid of asking for repetition or just saying that they don't understand in fear of being known as incompetent and it's good to remember when communication problems or issues appeared it may be not related to technical abilities of a person or just a simple basic communication problem related to language when I start my work at Red Hat I get to know that I will have a meeting normal stuff but in return on IRC I was so shocked I couldn't even understand how is it possible that the meetings are written and also I thought that IRC is a communication method which appears only on XKCD mems but it's still alive yep and now from the perspective also IRC is our official channel communication for open stack community so I understand why we are using IRC it has its advantages because it's very easy to use the whole community at that time I was very shocked and even prepared for that and I can say that Fedora did it in a nice way because they moved for some kind let's say more modern communicator because they moved to Matrix which is very nice web web user interface and it's easier for anyone from outside to just use it and the challenge of communication is to just choose a platform where anyone who would like to have an influence for the project will be able to use it communities are relations communities are people so building a relations inside a community and on the way community and commercial world is very very important and this is a huge role of community architects to just build these relations and events like that are very important for the communities because this is a place where you actually go meet people exchange your experience exchange your ideas and that's why how community is growing also this is very important to inside this community build a welcoming culture because when you are out of the communities it's very nice and I experienced that that you are just welcome here we are happy to have you here it's very good for the beginning also it's very important to have a safe space to ask to not be afraid to ask and grow and solve your issues the governance of the project the world of corporations and the world of open source differs in corporate world we have words like tickets KPI, performance, measurements you know it and at the same way they are not existing in open source communities it's just not the way they are working and this is a challenge to find the way to find the balance between these two approaches and the workflow in those projects the way how you are making decisions the way how you are making even the meetings the way how you ask all this all this issue to be solved they cannot be the same way in community like they are done in a corporate because voice of the community has to be taken into an account while making decisions or while doing just normal basic duties I have some teams for other newbies and fear most important use the time which is considered as being your training or the onboarding process as your time to be you know safe to ask because everyone are expecting that you will be asked that you will don't know but you will looking for answers, looking for solutions and it's normal and that's okay and you should take best of the time but you even can trying to ask proper and valuable questions you have to first try to do something so if you want to ask somebody a question don't let anyone to answer you with let me google for you or here is a documentation just read it try to avoid that because you know it's a waste of time and I think kind of lack of respect and first just try to do the job okay it's normal that you probably will faint or probably stuck on something at least try at least do some effort to solve the problem and this is something which will improve your skills make friends talk with as many people as possible because you will never know with whom you will collaborate so this is onboarding time this usually one one or two points depends on the company or community is the best time to meet new people to make new friends like saying hi I'm new here and just make networking between people and as I said if we only have a possibility to find a mentor do it because there can be a mentoring program in your company or in your community that you even don't even know about it so maybe you can just ask your manager or just try to find one because you will only be profitable out of having a mentor tips for mentors and as I say keep your documentation up to date I think I argumentated quite enough so I won't repeat that and again this world of asking but now asking from a different perspective you know why asking is so important because this is a route of communication so people should the communication should not be only in one way should be in both ways no matter if it's your mentor or your manager you should just exchange your communication I know this is very trivial as I'm saying but what I'm observing people don't talk so they're just sometimes they are not even aware about the problems of each others and if you are a mentor I mean when you are in a situation that you are you give some tasks to your newbie for example read this documentation or try this tool after the time ask your newbie and try to figure it out how is the level understanding of the newbie what did this newbie got from this reading got from this tutorial because this is a great way for you to understand how the newbie is looking at the project and also this opportunity to indicate the moments in your project which are overly documented or unclear or just undocumented because you as a mentor probably you don't have this perspective because therefore so long in this project that too many stuff are too obvious for you so this is an opportunity to you to have a different perspective on your project and what's important if this is asking is that this is not an exam not any kind of assessment because your newbie can feel a little bit in danger in this fire of your questions just let them know that you know just I'm asking it's not assessment it's important because without it your relations can be distracted give tasks to help understand you know when you are not only give a documentation to it but also give a small task to use this new stuff you just learned when I was learning our delorean tools our very versatile tool for building packages I got a task hey just try to build this one package with this tool and it's completely changed my whole perspective of reading this documentation because I was looking for some concrete information in this documentation which will allow me to solve the problem I've got and I think this is a very valuable way of teaching new stuff, learning new stuff give away background not only explain the solution of problem or a bug but also show how and why this happened this is an opportunity to explain workflows and actually the architecture in the project because it's actually having an anchor if a problem because newbie meets something did some some effort to solve the problem so they are inside the context so it's very easy to make notes about it and explain the context around it and hold your horses don't try to explain everything at the beginning because I'm sure you will have to repeat it later and general tips for a good start for anyone have a definition of them this is something I really like to have I mean that when you are making a description of a task in some kind of tax management define input and output of the tasks in a clear way describe this task that way if you for example go for a holiday and someone will do the task after you it will be totally clear for this person and I think this is a good description enough good description don't assume that somebody will understand what you mean without this description that way it's important and guessing is never an option because there is a very high probability that you will don't get what you expected describe the role when somebody's new is coming it's very useful to define what this person will do after the onboarding process and it's like setting an aim and setting a target for whatever they did and when the person will go through this onboarding process will be very sensitive for information which will be related to the job which they are going to be inside after the onboarding process for example let's say if you are a mentor of a manager let's say at the beginning this person will be in charge of developing our infra or in charge of new CI jobs and you know who hold this onboarding the newbie will be preparing itself for taking these responsibilities show collaboration with other teams or projects as I said it's very useful to know where to look for help and being able to ask it of course and who is responsible for what and the onboarding process is a great time for that and last but not least give yourself and themselves a time as a mentor be patient and don't judge people have a different way of thinking different talents, different approaches to problem as we are trying to build a diverse environment in our works also be ready that sometimes you will not only develop someone on a technical way or teach someone these hard skills that you will have to teach also these soft skills like problem solving abilities or doing dependency or required management or just way of speaking or discussing be ready for that as a mentor and if you are a newbie don't expect that you will be great since every first day any first day or even a first month because you don't have any idea about the complexity of the problem or the projects and don't care too much about the failures because if you failed it means that you tried and trying is something which is most important for make you growing thank you very much for your attention I'm here for you and I would very appreciate your feedback and if you have any questions I'm here for you Hopefully you will and I was about to suggest write a documentation of a couple of polls in the app or it you mentioned but I can see that you mentioned on the other hand what you are thinking about is that a mistake is it a bad idea because it just seems someone can do these things so what you are thinking about you mentioned that for the mentors and for the project it doesn't have good afternoon documentation and you mean by giving a task for making the documentation up to date yes okay okay so the question is if it's good to give a task for a newbie to fill some gaps in the documentation right I've got a task for that and yes I think this is a good task but don't be too too difficult in that don't give somebody a task to explain something which you don't really understand but I think it's a good idea for start yes okay the question was where to stop when to stop digging inside the project and stop trying to do yourself and actually ask for help well I've got this issue on my previous job that tried to do it but don't do much and I heard from my manager that you know it's up to you this onboarding process is for you and you will decide when you want to stop and move to asking because your manager or mentor is not inside your head and they don't know when is the moment you will feel frustrated or you feel stuck it's up to you if you feel that this few more hours will solve the problem go get it it's onboarding process of nobody you'll know track your hours but at least should not do it but if you are feel stuck and you feel stuck that even these two days more will not solve the problem just leave it, just ask because there's a waste of time and your energy so we have programs so somewhere as well of you know those programs you cannot say that there's no way for the newcomers to do the project but at the same time you have a very hard point because I can relate from my experience that this is the problem so now from your perspective what is the reason on one side we have those people at the university some of them who are teachers then we have programs that are aimed actually to having teachers and somehow it doesn't work so what are our okay I have no idea how to make your question short for a recording I try to understand with a context so remember that programs you are saying like this gcloud these are big corporations and most people what I was thinking about open source is this small initiative is just like I will start and do it or Google is not widely recognized as an open source company maybe let's say Fedora is because even though it's the same also supported sponsored by Red Hat and I would say people who are have a clear idea of joining open source they will not first think it will not be Google or programs of big corporations and from the other side from smaller projects it's very hard for me at least to find this way to join also this is another another threat here that people for example I wouldn't believe that I can handle the tasks in totally remotely without not knowing people and just using my keywords to solve the problem and actually providing a value so maybe I think people this is a mix because people doesn't recognize Google or order maybe Red Hat but other big companies as open source possibility and from the other side they don't have an easy way to start with smaller projects it's changing I have to say since I joined I have to say that it's changing I think for example there is a lot of programs to start as you're saying but I think this is a matter of time to changing but people are aware that inside open source that they need newbies and I think this is an important change here yes it may be branding but you have a Google in the name does it really have a branding of yes it can be if you know Google Summer of Code something Summer of Code like open source Summer of Code they won't agree but you know if someone is to hear for example program for open BSD it's very clear that this is open source but if you hear Google Summer of Code you understand okay okay come to me after thank you welcome to my presentation about Linux distribution collaboration on the mainframe yes I'm speaking about collaboration I don't watch any Linux distribution as a competitor anymore we have got a working group at the open mainframe project for all Linux distributions and therefore I will speak firstly something about myself then you will receive an overview about the mainframe and the open mainframe project what is that then we are coming to the Linux distribution working group which is part of the open mainframe project and I will tell you something about our goals for all Linux distributions and upstream projects now I would expect all here are available because you are pro open source communities we are upstream working together and therefore we have decided to include also upstream projects now that we can collaborate better together and then we are coming to support you as developers how to that you can receive support in development and architecture specific stuff as an example the S390X architecture and what is the Linux one community cloud or the Linux one open source of the community cloud that you can receive VMs for free on our mainframe as developers something about myself I am Julia Krisch and I am working as a lead IT-OT software engineer at Accenture I am doing that part-time because I am also a part-time master student in computer science at Friedrich Alexander University in Erlam Nürnberg in my free time I am an open source contributor until 11 years now yes 11 years because I have got a education in computer science as a computer science expert and I was a student with work experience because I have worked as a Linux system administrator then and now I am a member of the release engineering team I am responsible for the S390X architecture at OpensUser and I am a team lead for OpensUser C systems I raised that I want to found the Linux distribution working group together with one IBM therefore I am also one founder of that and I am speaking for all Linux distributions who have joined to us what is the mainframe on the right side you can see all sizes which are available from IBM the smallest one here is the latest release from April that is a rock mounting system and the mainframes are well known as large high performance computing systems especially used for banking systems, entrances and everywhere where you have to multiple transactions in parallel now it is also available in a smaller version and if you are speaking about architectures they are also well known as big indian systems and the architecture is the S390X architecture it is used for mission critical data like banking systems as I have explained and you can run thousands of VMs on such a system from that we are coming to the open mainframe project the open mainframe project has been founded in the year 2015 and is under the hood of the Linux foundation the focal point is the deployment and usage of Linux and open source in our mainframe computing environment and yes we as Linux distributions said if you have got also the usage of Linux as a focal point then we want to be integrated because if you can see mainframe centric projects under the hood of the open mainframe project more COS specific or generally C-specific you can see different projects here which are used mostly for COS and not Linux some of them are for the improvement of the command line where you can connect better and where it should be as an open source software so we is for better deployment of Linux and everything else on CWM but I want to highlight one thing we have got also a mentorship program therefore if you are interested for mentorship in the area as a mentee or a mentor we are supporting also the development of open source projects for the mainframe for Linux especially and therefore there are many projects under the open mainframe project then we are coming to the hot topic working groups there are free working groups available I have joined two of them the Linux distribution working group exists since two years now the modernization working group I have joined as a Linux contributor but the focus of mainframe modernization should not be only COS related I want to have also a Linux background the inside of that and then there is also the cobalt working group available which is also responsible for the programming course at the open mainframe project we have started two years ago shortly after the IBM CD with a kickoff with community Linux distributions because I and Elizabeth we said we don't want to have one a Linux distribution for the mainframe we want to have support and infrastructure available for all of us and especially IBM said in the past we provide only official support for Rails less and Ubuntu we said we want to have also really good support for all community distributions therefore I have reached out to Fedora Dan Horak has been available for Fedora when Elizabeth went to Debian and we have received Ryan for that and with that we created our Linux distribution working group now we are more afterwards in the next step we said we want to include all enterprise Linux distributions then Suze has joined in the next step Ubuntu has joined in the next step and Dan Horak said he wants to be the representative for Fedora and Red Hat in our Linux distribution working group therefore to the Red Haters if you are interested for speaking for Red Hat and supporting Dan on the other side you can join us in person Fedora and Red Hat have got the same split as Suze and Open Suze one Linux distribution for one person should be enough but anyways that was the next step to integrate the enterprise Linux distributions and then after my talk had forced them in the year 2022 they have joined in our Linux distribution and Rocky Linux have joined also but if any other Linux distribution wants to receive support for building for S390X I can join our structure is we have got our founders as co-chairs I am a manager Elizabeth is Ubuntu member therefore she has been the second founder from Ubuntu's side as an IBM employee now she is also the head of Open Source program office at IBM therefore we have got the highest level at IBM for Open Source software within our Linux distribution working group then we have got one representative for every Linux distribution we need that for the input and that we can share our knowledge whether we can fix something in our Linux distributions if anything is happening and our Linux distribution has got a solution and additionally on top we say also upstream developers are welcome during our meetings and our sponsor has been through at the beginning our goals at the Linux distribution working group is to create a place to collaborate across Linux distributions we are an open mainframe project mailing list, wiki and chat yes we have got a wiki we have got a chat we have got a mailing list mostly used the place is the mailing list I have got a problem can we fix that can we receive support and mostly you receive support within two or three hours from IBM site or from our Linux distributions and that is providing also a space for distributions to request for help on the S390X port then and then a topic from Elizabeth to ensure any and all infrastructure which is required is available for supporting the ports that is the mainframe in the background Red Hat and SUSE have got mainframes Canonical has got a mainframe Fedora is using the Red Hat mainframe, Open SUSE is using the SUSE mainframe and Debian has got our own mainframe Alma Linux and Rocky Linux have received also infrastructure but in case what I want to give as an example to this goal is that nobody on Debian site from the infrastructure sponsors wanted to be responsible for the mainframe anymore and the CI CD pipeline as an example didn't work anymore and that was the reason that we said okay we want to transfer all the pipeline stuff into the Linux one community cloud that IBM can be responsible for the infrastructure for Debian continuously therefore if anywhere we are shut down of our mainframe our Linux distribution working group has got this goal and therefore you will receive a new mainframe or you will receive a transfer of the mainframe or the community distribution as an example we will receive alternative mainframes where we can receive infrastructure for pipelines and buildings then the topic what I wanted to have within it is the better support from IBM to fix S390X specific bugs in the past that has happened mostly for enterprise distributions only now you can receive a support continuously and really fast I would say I want to create a bug report because of a kernel problem I am receiving a bug fix within one day now that is a really improvement I would say therefore if you are writing to our mailing list or sending a bug report to the program managers at IBM you have got a really fast response there now from that we are coming to our collaborative process we want to collaborate and support each other we don't watch us as competitors anymore and we are doing our problem discussions on our mailing list as a first step we can reproduce issues sometimes in other build pipelines if anything is happening we have got open discussions on the mailing list about problems and then we are forwarding issues and ideas of improvements for IBM as an example through the kernel developer wanted to receive an improvement in the tooling suite for the kernel configuration distinguished engineer Ulrich Weigand has interacted a nice idea forward and within two weeks they created a patch for that therefore yes the distinguished engineer Ulrich Weigand is also available on the mailing list and in our discussions and then we have got collaborative small projects I will bring in on the next slide the example with OpenQA but anyways we have got also monthly meetings which come together with a review what has happened in the last month is all resolved what has been happened on our mailing list as an input and to do tasks and then a discussion what we want to do next and whether we have got new releases and everything therefore we are reviewing all and we are creating new goals for the future there one small project what has happened is also the S390X test contributions for OpenQA some of you will know OpenQA is a testing suite from OpenSUSE which has been used also for Fedora Debian and Gantu in the past and of course it is less and IMA Linux has joined using OpenQA and said we want to enable the support of Red Hat KVM in OpenQA where it did that we have announced that on our mailing list and Rocky Linux stepped up we want to use it also we want to integrate also our contributions therefore we are now OpenSUSE less Fedora, Debian, IMA Linux and Rocky Linux and Gantu and we said afterwards contributions to our general OpenQA repository on GitHub is of course welcome therefore it is a good point that we start also a collaboration for OpenQA that all enablements and test suites are coming together and every Linux distribution can use available tests that should not be only for S390X we want to have it also for X86 and other architectures finally should be Linux distribution specific stuff where for the needles screenshots are different perhaps distribution specific configurations from that we are coming to the topic are there any problems during your development process for S390X because we want to be the point of contact for all Linux distributions if anything does not work on the mainframe during the process problems are affecting mostly all Linux distributions and you as developers also and not only one single person or one single Linux distribution we are receiving fast support via our mailing list of the Linux distribution working group we can build issues sometimes we have got situations that only one Linux distribution is affected but that is mostly based on the kernel version that Ubuntu as an example is using a special older kernel version or open through the tumble read is using the latest kernel version the same as Fedora is using another kernel version that can happen that something will break based on the kernel but that will be also in our focus and we will support there but mostly that should on application level everybody the IBM distinguished engineer is responding on development problems when I have forwarded also upstream project issues where the developer didn't know how to enable the software for S390x anything small didn't work I have forwarded and the fix was available in three hours all were happy then that is also possible therefore if you know anything related to that you can forward it to our mailing list with the WG Linux distros of the open mainframe project org and afterwards we are receiving solid solutions also for upstream mostly within hours or some days if you want to know something like that also for ARM I know that Linaro as an example is providing such a mailing list for ARM specific problems for all Linux distributions when we had email discussion about that and security related there exists mailing list for all Linux distributions therefore you can find something for mostly all architectures already now you want to I expect you want to know how can I develop on a mainframe without having my own mainframe at home or in the company where you can find the Linux one open source of the community cloud and the Linux one community cloud you can receive a IBM for 120 days for developers, students and professors in the Linux one community cloud you are creating your user account and you are receiving access the account will be removed afterwards and recreation is possible but the best case is you can receive also as an open source project long-term access in the Linux one open source software community cloud you can use this virtual machine request write down any reason for what you want to use it you can define which size of memory you and everything else you need and when you can receive it I as an example I have received C70Ms for development and hackathons for our open-through-the-sea systems team and the single thing what you need is any open source project as a reference now we are coming to the last slide finally I can say collaboration we can watch as a benefit upstream contributions are available for all we are lowering the research and development costs that's valid for IBM the community and every open source enterprise company we are receiving the same solutions for all Linux distributions we are sharing our knowledge between the communities and finally all are happy and are receiving the fixes faster and then we are receiving diverse community ideas every Linux distribution has got another thinking has got other ideas and with that you can increase also innovation and on this way we can accelerate the Linux development for S390X now we have got some minutes left for questions yes you can use QEMU already as an emulator as an example and there is also Hercules available but IBM does not really support Hercules I would say but we have got some people are laughing I wrote my bachelor phases based on QEMU IBM has also got a KVM team related to that I can say but if you want to develop and building stuff I can really recommend these two links if you want to use it only one time you can use the 120 days version if you are Fedora contributor of anything else you can order for yourself via whistling the hint is that you can only choose between Ubuntu, Rel and Slash as a foundation therefore you have to choose one of them and if you want to build on Fedora when you have to use KVM or something like that Slash is possible that you can upgrade to OpenSUSE where we have got a method where we can use Zipperdoop OpenSUSE repository and we have got OpenSUSE in our VMs I am working on it that I can receive IBM to the site that we want to support also community distributions that Fedora and Debian can be also integrated where but it seems that it needs some time because that is not officially supported officially supported therefore I can recommend using QM for S390X and when you should have got a good emulator that has been improved within the last 10 years that's the best choice related to emulators they didn't like it really much but my bachelor thesis was about something like that related I should create container images build it and run it integrated into QEMU for emulations and that was when transfer from Kazan Reisen example exported to X4 and when the emulated VM should start on X86 that was an interesting bachelor thesis project I would say you can see also my bachelor thesis and my name is bachelor thesis on Github but yes they are working on it it was not really wanted in the past but IBM wants to become open source therefore they have improved that and this method is the method they want to provide open source projects who don't know anything about mainframes in the background of Github or something like that IBM wants to receive enablements of open source projects that are not responsible for that therefore anyway it has to be enabled and on the other side they want to be responsible for the foundation, kernel development KVM network devices and everything else and therefore you need a solution where you are supporting the application development on community side and stable foundation on IBM side I don't want to fight with it really and also I have some family stuff so I decided that's the other one I'm planning to do it at it will be closer like when it was in here in Hungary and everything it's easier so and also the budget is quite high yeah I was surprised our whole team is getting to a flop which I was not expecting to happen I don't know why our budget worked out that way alright we got some people yeah it's just so okay hi everybody I'm Adam Williamson I've been working on Fedora QA for quite a long time now since about 2009 and this is Miro Vegkerti who is helping out with the Fedora CI parts of this talk yeah I'm going to kind of get right into it because the slide deck is pretty long so I want to make sure we get through the time Miro do you want to do a quick personal introduction yeah so I'm Miro I work mostly on LCI but we have some infrastructure also in Fedora and I'm a Fedora contributor we are providing some infrastructure for running the testing the other testing form service so yeah what we're aiming to cover today is it's not going to be a deep dive into the technical details of the systems what I want to talk to people about is what are we actually testing what is this achieving and for people who are Fedora package maintainers how do you interact with it and how is it helping you so let's get going a little bit of quick background how do we test an operating system this is a huge topic it's a challenge deciding what to test when to test it we have in the time cycles we operate on so there are four levels right now at which you can really do testing within Fedora there's package source control so Fedora packages are stored in a Git repo and you can do testing on things that happen within that repo pull requests, commits anything you want to do with that you can do testing at the level where builds happen when you do a single package build within Koji we can trigger test on that you can do testing at the level of the update which is the sort of abstraction we have for getting feedback on single builds of packages or multiple builds of packages before they actually go into the distribution and we can do testing of composers which are when we build the whole of Fedora together and actually produce the images, docker containers disk images, whatever you like that we ship to users we can test the composers so the background, manual testing ye oldie days so this is up to about 2015 which is going to get covered on the latest slide back in the days before we started getting better at this we basically didn't really do any testing at the diskit level just maintainers who were really keen might self check themselves somehow they might have some little script they run on it but there was nothing automated the Fedora QA was not really doing anything at that level at all build testing, all you could really do is have a check section in your package which many packages do and it's a good thing but it's not really sufficient and that would just usually it just runs the unit tests from upstream update testing once Bodi came in the point of Bodi originally really was to allow for testing and feedback on updates so that packages didn't just randomly go into the distribution and break stuff but before we started doing automated testing it was really just down to people doing it so maybe if your package is really popular, if it's Firefox you're going to get a lot of feedback if your package is not that obvious to users even if it's really important it may never get any feedback at all it was very haphazard maybe this build gets tested but this build doesn't and what actually got tested from build to build again was just kind of down to people so it was again it was very varied it was very inconsistent we didn't really, it's very difficult as a manual tester to test whether an update breaks other things and especially whether it breaks the system we had almost no testing of that and compose testing was the thing we mostly focused on I guess and we were getting pretty decent at it doing it manually we had a whole process for it which I'll cover soon but it was very time intensive anyone who worked on Fedora QA back then my life was just booting up 6 VMs every day and doing install tests and wishing I was dead it was not much fun the amount of work we put into it as humans you can't test every compose we would test some snapshots every cycle and then the release candidates so we'd test I don't know 15, 20 composes a cycle but there are hundreds each cycle so we were not covering very much at all little history of how validation testing worked I'm a history nerd so I love this on the left here this is the earliest recorded formal validation testing of Fedora in the history which is Fedora Core 5 and this is a wiki table and there were 26 tests in it the two people did almost all of them as a test I'm just counting a row I'm not counting the environments because counting that is harder and then up to Fedora 21 which was 8 years later I think we were still using wiki tables but they're much shinier and more color coded and we had got up to 138 tests Fedora 21 was the last release before we really started doing automated testing so that was you know as far as we got with humans doing all the testing did I miss anything on my notes here yeah so we this is like six times as much work as we've been doing on Fedora Core 5 but it still wasn't enough we weren't covering everything we were missing bugs we were taking too long to catch things we would often only find something when we got to beta it had been broken three months earlier and tracing that out was a nightmare and just as a note on the test counts Fedora 39 now has 202 tests so we've continued to kind of scale the amount of testing we're doing that way so that was you know that was as far as we could really push manual testing we were at our limit at that point so we came to the conclusion that we need to do automated testing what are the automated test systems that exist in Fedora right now the key to I'm going to cover other things very briefly later but the key to our Fedora CI and OpenQA so I want to do a little bit of explanation here one of the questions we often get is why are there two you know automated test systems in Fedora and this has been a process and a history and there's a lot of nitty gritty in the background but we've come to a pretty good understanding of why this is and why it makes sense and why we had them Fedora CI the point of Fedora CI is really to provide CI type work flows infrastructure processes to people operating in the Fedora environment Fedora CI the people who work on Fedora CI don't wake up every day and say is Fedora broken they wake up and say are the tools and processes we're providing to Fedora people are they working are they good tools how can we make them better that's what Fedora CI is about providing CI services to people working in Fedora OpenQA is there to do testing of Fedora OpenQA is literally the automation of all the stuff the Fedora QA team was trying to do manually and really burning out that's the point of OpenQA OpenQA is there for us to be able to figure out if Fedora is being its best possible Fedora it's not a service we're providing for other people to write tests of their thing in the Fedora environment that's not what it is so with Fedora CI the system is the point that's what Fedora CI is systems and processes and you know interconnections and it's the job of Fedora CI is to provide testing systems for you all to use with OpenQA the system isn't the point we just use OpenQA because it was a thing that was there that worked to do the job the point of OpenQA is helping us figure out is Fedora broken who broke it what broke how do we fix it that's what OpenQA is for specifically OpenQA is specifically not about letting other people come in and run tests of their thing we kind of thought about going down that road several years ago and we've specifically decided against it you can contribute to OpenQA but it has to be within the context of what OpenQA is there for so if you want to contribute something which helps us test whether Fedora is great perfect but if you want to test your little widget in the Fedora environment that is what Fedora CI is for so that's kind of the distinction between the two OpenQA was started by the QA team in 2015 as a Skunkworks project was to automate our stuff Fedora CI started in 2017 and it really came out of we try to be transparent about the relationship between Red Hat and Fedora Fedora CI really came out of an effort within Red Hat to reduce the distance between Fedora and REL where it came to testing automated testing because how we do things in Fedora is completely different from how we were doing things in REL at the time and Fedora CI is part of a coordinated effort to try and make it possible to share more things across REL, CentOS, Fedora so a large part of Fedora CI is making it possible to take things we have within Red Hat that were never applied to Fedora and make it possible to apply those tests to CentOS and Fedora yep, so I am going to go Miro will talk about Fedora CI after I've done OpenQA I'm going to go into the sort of details of what OpenQA is and what we're doing with it so what OpenQA it's an original strength what it's really good at and what is really cool is that it tests like a human like I'm not going to go into details of how it does this but OpenQA is effectively running a virtual machine looking at what's on the screen looking for little areas of the screen that it expects to see clicking on them it can also type things so it can type commands, get the results out and so it's great for automating manual QA testing because that's what we did most of the time we span up virtual machines and clicked on things it's also great because it doesn't care about the operating system at all you can use it to test Windows, Linux you can use it to test firmware interfaces all it needs is a VNC connection to the computer and a way to click on things so it is really appropriate to what we're doing and as I said what we're trying to achieve with OpenQA is high level functional testing it is not here for unit testing it's here to tell us is Fedora broken is something that we care about something that is part of what we want Fedora to be for people in the world is any of that broken that's what we're doing with it so what do we actually test with Fedora we run a subset of tests I think there's about 200 of them on every Fedora compose also we're going to subset on CoreOS composes IoT composes, Cloud composes anything that's a compose in Fedora gets tests run on it we also run a subset of tests plus extra tests about 50 to 60 on critical path updates for every branch of Fedora stable branches, branched and worldwide that's what gets the data gets the tests run on it if no one knows what critical path is it's really just the set of packages that are the most important and this limitation is purely resource based I would love to test every update that goes out but we just don't have the capacity for it so this is what the OpenQA WebUI looks like and you if you're not part of the Fedora QA team you actually don't need to look at this very much but I wanted to make this talk practical about what things are looking at so on the left hand side over here this is just a subset but this is a view of some of the tests that were run on this is a compose from a few days ago these are most of the tests that ran on the silver blue installer image and down the bottom you can see one of them fail rpmoistree underscore rebase which does what it sounds like it tests rebasing rpmoistree type install on the right here these are two different screens on the right here this is what that fail test actually looks like and you can kind of see there's a bunch of screenshots when it's got a green outline it saw what it wanted or a command did what it was supposed to where it's got a red outline it didn't see what it was expecting to see and you can click on each of these and you get a full view of the screen and you can see like the top here is it's in emergency maintenance mode when it was expecting to be a booted system you can see what you get with the OpenQA Web UI and this is a real thing you can't rebase from raw height to 38 at the moment because of I think an SE Linux thing so that's how that works what do we actually cover with the OpenQA compose tests? we cover 75% of the validation test suite which is the set of tests that need to pass for a fedora release to go out the things that aren't covered are things that are very difficult to automate or things we explicitly don't want to automate we always want to have a human in the loop so we always want a human to test that the images actually boot and install on a real computer, a real container a real virtual machine just in case the automating testing system missed something I never want to release an image that no human has actually tried to boot up so we can't really do 100% but we do our best we also cover some stuff that's not release blocking like we test silver blue technically release blocking but it's very important so we want to test silver blue a lot of the tests are install tests we really exercise a lot of the install we install a bunch of different images we test different package sets can you install known, KDE minimal, a bunch of different ones different partition layouts butterfs, XFS, EXT, LVM fin partitioning whole laundry list of different partition layouts languages, we test English Arabic, French, Japanese and Russian to cover a kind of different fonts, different symbol styles and also right to left which is very important we test different firmware types so we do installs on UEFI and BIOS to make sure they both work but we don't just do install tests OpenQA started out mainly doing install tests but these days we do a lot more we have the base tests are the core operations for any Fedora install, can you install a package can you remove a package, can you update a package does logging in work, does logging out work, does rebooting work is SE Linux in enforcing just in case we mess that up some time hasn't happened yet but it might can you start services, stop services restart services, enable services disable services, all of that is tested system logging is logging working can you get the journal upgrade tests, we test upgrade of all the release locking package sets workstation, minimal, KDE server, we test them N1 and N2 so we test 36-38, 37-38 both of those and we also do a test that a free IPA deployment can be upgraded an entire deployment server replica client upgrade them all make sure they're still working after the upgrade so that's one of the more complex tests we have graphical desktop and application tests we test Nome and KDE a lot of their core functionality the overview, the user menu the stuff that you expect to work on a desktop we test that every installed application at least starts up and stops, trying to test of them all functionally is a lot of work but we test that they don't just crash or just not launch at all on Nome, my colleague Lucas I don't know if he's here anyway, he has been working on writing a bunch of Nome application tests so right now we test about 50 Nome applications in quite a lot of detail like the maps application a bunch of stuff, can you look up a place can you do a route real proper detailed functional testing of the apps we test desktop login which isn't just login but also can you log out, can you switch users can you lock the screen can you unlock the screen can you reboot from the login menu that kind of stuff we test that def notifications work which includes testing that on a live image you don't get update notifications we test one, we test printing using virtual printers obviously and we test upgrades updating and upgrading yeah so that's pretty intensive and we test all those tests gone on silver blue and we also run all those tests on an upgrade as well so we upgrade workstation and then we run all those tests to make sure they still work on an upgraded system we have a bunch of server functionality tests like the key features of server free IPA, a database and cockpit, we do a bunch of functional testing of those do they work, do they show what they're supposed to show so for every single compose we are testing all of this stuff like every time a nightly compose comes out all of this gets tested update test coverage we don't run all of those tests on every update again capacity issues but we run about 50 to 60 tests and this has got a little refine recently because we're doing stupid stuff like running all the gnome tests on KDE updates which was just idiotic so I did a lot of work to split up the critical path into groups and that allows us to only run appropriate tests so not every update runs every test anymore but there's a total of about 60 tests we do the tests I talked about on the last slide the KDE workstation and server tests not all of them but most of the key ones again a subset of the base tests and the desktop tests free IPA, cockpit and database tests I think we've run all of them pretty much and then one thing that's really important and I'm pretty proud of is for every single quick path update we're building a network installer image a gnome live, a KDE live and a silver blue installer image which was a lot of work and making sure you can do that build you can run an install and the install system works so what this is really testing is that the update doesn't break the compose so we know that once we push this update we can run a network which was really key um and yeah again just a key thing to note from earlier on when we're testing an update to you know libfoo we're not testing is this libfoo the best libfoo it can possibly be we're testing does this libfoo break fedora and this is always the goal for OpenQA all of the testing is in the context of making sure fedora is okay so yeah I mentioned this already but the main limitation is really just the tests I would love to run all the tests on all the updates I would love to go across arches but we just don't have the machines if someone wants to give us more machines that's great I'd also like to thank the meta folks because they're planning to hire a contractor to work on cloudifying OpenQA which would be a great way to get you know more test resources so I'm hopeful we'll be able to get somewhere with that soon so scale and success just a perspective on what we're actually achieving with this we have two instances of OpenQA in staging we test more arches we have PowerPC in staging right now we run over 100 tests at a time on each instance I think we've run over 3 million tests since 2015 discovered hundreds of bugs 358 is just bugs that are tagged OpenQA in bugzilla but that's a huge undercount because I only started tagging after a while and a lot of things we just fix it because I'm improving packages so I just fix stuff or we file issues upstream so it's a lot more than that in a typical day just a typical day, not a busy day it'll be one or two fedora nightlies depending on where the branch exists there'll be 3 core OS or cloud composes and we'll test about 20 updates on a busy day you can double or triple those numbers just some examples of recent bugs that OpenQA has caught right now there's three failures that we know about from the EFI system partition size increase thing which I've filed in the past I've noticed that for another two months Firefox just crashing on startup which it does so we caught that and got it untagged so that doesn't affect how it uses the Arabic translation just disappearing from the installer that was a fun one to catch exactly notifying about updates when running live that started happening and we caught it and that's just the first page of search results I just pulled up the first page of most recent results from bugzilla and that's what's on there another example that happened recently from update testing a new util linux build it was mounting the group partition read only which obviously makes the whole thing not work and in the past that would just have landed in raw hide and the next day people would have been like hey my raw hide system isn't working but instead we caught it we got the util linux untagged nobody saw that except people who were running the test OpenQA resources yeah this is the thing I mentioned earlier but it's important to make a service we provide to fedora the testing we don't necessarily want packages to be trying to debug failures themselves so we don't just run the test and run the system we investigate the failures and we make them actionable ideally we fix them if not we at least turn them into a useful bug report so you don't have to look at the OpenQA results and try and fix it yourself so that's the philosophy about OpenQA if you need to contact us we have a mailing list fedora chat room I should mention OpenQA is originated by the SUSE folks it's a great system we're really happy to have it we collaborate with them thanks to them for all the work they put into it and that's the upstream site for it there's a wiki page which kind of explains the fedora deployment and this last point I added after the keynote I really love the keynote OpenQA is an open source service everything is open not just the code but all of the stuff that we've written around it the tests, the scheduler are made via fedora infrastructure Ansible scripts which are in a Git repo and you can contribute to and you can even do a pet OpenQA deployment using those Ansible scripts and it should mostly work so you don't have to contribute to OpenQA but if you want to, you can and I'm now going to hand off to Miro to talk about the Fedora CI so hey so if you want so Adam has less work than onboard your tests to Fedora CI basically you do your part that your software is stable and then will have less work and he will be glad so Fedora CI is here for you that you as a contributor to Fedora can do your part you can stop basically your build which sometimes breaks its software you can stop it before it enters Fedora if it enters Fedora, it's on Adam that's the story so what can you do in RHEL because it has similar similarly there is a Jenkins instance which is actually calling testing farm but my team takes care of it's fairly stable and it works well you can see the documentation here at the docs there is also a nice how-to guide to get you started, quick start guide really copy pasting stuff and you can add your test whatever that is in Fedora there are two places where you can run tests first is so you can run tests the tests are always placed in this Git repository you just drop there some files the tests don't need to be there they can be leaked from GitHub wherever you want that's thanks to TMT because it can share tests with upstream repositories and so on so it's maintained by you it's your responsibility you can of course ignore the results so you can open this Git if you have that test there it will be reported but it's not gating you can just merge it if you want then you will break maybe Fedora so yeah it can be made gating like I think Zul has capabilities for that so that's the first place open a this Git pull request nothing gets yet to Fedora test will run, you will validate it that build that was built from this Fedora and before breaking Fedora you can you can fix it you can also run tests from other components for example I would love if Selenux would run cockpit test suite because cockpit touches a lot of components and so on so it would be great if the great cockpit test suite would run for Selenux and then maybe let's work later on for them so yeah so that is one thing then also the tests are run for validation after merging so once the stuff gets to row hide there is no gating again so on BODHI you can again see after the production build was built you can see the results on BODHI so that's the second place where Fedora CI runs and also OSCI team which is the second team who is running this stuff is running some generic tests installability RPM-inspect and RPM depletion they really try to make sure that those packages are installable without issues and the RPM-inspect make sure that the sanity of your RPM is fine that's right sure this is all I said before sorry I think like if you want to do it super smart do it be a packet on github or github if you are lucky that your upstream is there if not then you can do it on this git merging quest that's a better place and the worst place is after merging when stuff already gets there we would love to start gating on Fedora so your tests can be gating but currently they are not it's really on you to make them gating yeah I do want to just quickly highlight the packet workflow if people aren't familiar with that if you kind of really buy in it's a cool system because you you go from your upstream project all the way to Fedora in kind of one workflow so you have your packet configuration upstream and you have your tests upstream and then you can do all your development upstream packaging stuff kind of happens automatically and the tests get run on your upstream pull request on your spec file pull request it's a really nice integrated workflow with all the testing in the background happening via testing farm which is Fedora CI's back end so that's a cool thing that it's worth checking out have a look at packet I'm just going to blow through this really fast there are other things which kind of are automated test systems even if they weren't intended to be or packages build dependencies change it tries rebuilding it in Copa and tells you if it's broken which that's effectively a QA thing so it's kind of cool Fedora release auto test is written by our colleague Lily Nye, another person on my team and that runs tests on weird hardware that we can only really do in Red Hat Speaker because it's a Red Hat internal test farm thing so it can test weird enterprise storage things like Vybertanel over ethernet iSCSI and things that we can't really test in open QA so that's really neat so that automates those tests which were really hard to get done before RelVal which is a silly thing for reporting results manually to the wiki also actually when a release validation when a new release when a new candidate release comes out it runs the size checks automatically and files those results into the wiki so that's kind of an automated testing Fedora Core OS has its own entire like CI release workflow which is really cool and really integrated and it's very modern you know you're bolting all this stuff on afterwards Fedora Core OS was invented much later so they have a really cool cycle and they do all their own CI Google based CI is getting tests run in pager for your project and packet I just mentioned result delivery reaction so testing is one thing but you have to do something with the results so this goes back to that earlier slide about the four places testing can happen so diskit is like the earliest place you can get your test results in so you know your package is in diskit we have a pager user interface on the front of that which is you know the github style forge thing and your pull requests if you manage your package if you actually do pull requests which not all package maintainers do but if you do pull requests then your tests that are in your repository will get automatically grown on you'll get the results in your pull request as Miro mentioned so this is like the earliest point you can get it in and if you're disciplined to use the pull request workflow it's really useful to have all your test results at this point Bode the web UI of the package of the update management system is probably the main integration point for results this is where you're most likely to actually see your results I think for a normal packageer working in a normal way on the automated tests tab in Bode you get like a list of all the test results and this is you get your Fedora CI tests and your OpenQAI tests OpenQAI tests all together the ones that have Fedora-CI at the front come from CI the ones that had update at the front come from OpenQA but because the back end they're all kind of compatible you get all the results in Bode we've improved this quite a lot recently the first version of this talk had me apologizing a lot for a lot of the bugs in this view and I'm hoping people may have noticed it's got better recently we would have weird things where the sync between the back end and the front end of Bode was off so one would say your package was gated and one would say it wasn't and hopefully that's been fixed you now see running state so when a test is queued or running you will see it on this tab before it would just say the result was missing which is confusing but now it actually tells you that the test is running, wait a bit and you'll get the results so yeah, new queued running state hopefully more consistency right now updates for stable releases and branch releases are gated on most of the OpenQA test which means if one of those tests fails your update cannot go stable right now Rohide is not officially gated I'm probably going to turn that on after this talk, I wanted to do it during the talk but we're running out of time because Fesco has approved it I believe and that'll be a big thing but anyhow we've been shadow gating Rohide for a while which means when we find a bug in an update for Rohide we will untag it before it makes a compose so effectively Rohide has been gated for several months we just weren't telling anyone about it that's me and Kevin Benzy but it's been working great it's made Rohide way more stable packages can configure additional gating requirements so if you want to gate on the tests that are in your repository you can drop a gating.yaml file in your package repository and it will be those tests will gate there's a button for waving bogus failures if any test is failed and it's blocking the update there's a button that says way failures please only click that if you're really sure the failure is a bogus one there's another button for re-running the tests if you think the result might be a bogus one hit the re-run tests button and see if it keeps failing then it's probably a real problem or a bug we need to figure out what do you do if you see a failure in Bode as a packager if it's a pack, if it's a test that you put in Fedora CI yourself you should probably go ahead and debug that, that's your problem if it's one of the generic tests which are the installability, RPM, depth lint something like that that shouldn't gate the package by default but maybe you've turned on gating for it try and debug it if you can't then contact the Fedora CI team and they will help out open QA failures again this is part of our full service model if you're really keen and impatient you can try and diagnose the failure and figure out the details of open QA but if not just stay calm and wait and we usually I refresh the open QA web interface about 12 times a day, like I'm always looking and any time I see an update failure I'm going to investigate it I'm going to either fix it or I'm going to file a bug for you so you will get a report which you don't need to know how open QA works you will just get told what's going wrong and I will ask you to help fix it so if you need help send me the Fedora chat, the mailing list send me an email, carry a pigeon whatever you like and yeah, re-run the test if you're not sure if the failure is genuine, wave the results if you're really sure it's not a genuine failure for problems in Bode itself you can contact CPE the Fedora infrastructure team or you can contact me because I still have work on Bode too so I know how to do it and quickly, yeah we still use the wiki, like for 20 something releases later we still have tables on the wiki we still need to build the results in them the reason we do this is just because it's the only way we have to integrate automated and human results so we still need some manual testing for composers so people put their results on the wiki page and the crazy thing I wrote let's open QA, file its results into the wiki so when we're deciding whether to release a Fedora release we go and look at the wiki page and we check that all the tests are covered and all the tests passed that's why we still have the wiki I'm going to skip over this because we're short on time there's a lot of stuff behind the scenes here and this is all shared between Fedora CI and OpenQA on the front end there are different systems but the way they file results the way they all talk to each other is all kind of integrated and shared this, I really like this slide if you've been looking at cat pictures for the last 30 minutes but you want to say you came to this talk take a picture of this slide this is everything, this is the whole talk updates and composers tested by OpenQA we don't gate composers right now but we review the results we gate updates you can gate your pull requests in diskit and Cache does FDBFS that's what we've been talking about the whole time the future, again quickly, what are we doing with OpenQA we want to do more tests we're always writing more tests so cover gnome applications gate raw height updates I would really like to turn that on and I'm planning to soon we'd like to cover more arches that just need more hardware cool thing we're working on right now is doing bare metal testing in OpenQA which uses Raspberry Pi KVM it's a really cool project more tailored update tests so because we now have this grouping thing I kind of want to do let's run all of the installer tests when we have an anaconda update so that would be cool and maybe move it to the cloud Fedora CI plans, Miro we are almost out of time one thing, if you have STI we are going to get rid of it so migrate to TMT, there is a nice guide it's important stuff for all of these otherwise I think Fedora is quite stable if not they can reach me on chat Fedora project or Fedora CI we are there to help you and if you watch the recording of Miro's talk he has a lot more about all this stuff in it other plans, yeah one thing I skated over, the biggest challenge we have right now lots of other things go into making a Fedora compose the kick starts the comps pungy configuration, workstation OS tree config which is actually where all the immutable configuration lives when these things change there is no testing if you make a mistake when you are editing comps the compose fails and we just don't know about it, so I would love to do some stuff to do more testing of these but we got to have plans we made it through the slide deck thank you everyone for sticking with us by talking fast I think do we have five minutes for Q&A or is that the slot? five minutes cool, yeah so the question was can we also test Turkish yes we could I am loading a new language just requires doing quite a few screenshots but yeah we could we do have an issue tracker I'll talk to you after the talk with OS Auto in Distry Fedora is the project for the tests and if you file an issue on there I will file one, yeah we can definitely look at doing that cool okay yes, let's do one at a time so I don't forget, the question is can we test our suspending theoretically we could test it because you can suspend a VM and resume it but it's not very useful because all the problems that suspend and resume tend to be on real hardware so yeah that would be that's one of the reasons we're looking into the real hardware testing sorry I'm not sure what you mean, application cache files and debugging oh yes good question I kind of scourged it over Miro can talk about CI but in open QA if the test causes an application crash so there's a thing in open QA when the test fails it runs a post-fail hook so we upload a bunch of generic logs and stuff but we do also test for abort crash dumps and core dump control crash dumps if we find one it gets uploaded so there's a tab in the open QA web interface, assets assets and uploads I think and if you click on that for a fail test you'll see a bunch of logs and any crash record that happens so one of the things I do when I see a failure is oh there was a crash I analyze it and then I file you a bug with a proper backtracing I just want to say that maybe there could be a generic test for all packages for that like maybe but then the test would have to run it to see if it crashes anyway yeah more questions? I think you had your hand up first okay I think great question yeah so the question is how long do the tests take to run for Fidority I it's very complicated because it all depends on it's all in the test definition for open QA the compose tests I believe take about two hours to run completely for an update it takes again about two hours in total but most of that is for the silver blue test which isn't gating so for all the gating test to finish takes a little over an hour I think mainly because of the live build and install test which takes a while and yes there's a lot of parallelization so the way open QA really works is you have these things called worker hosts which are systems that spin up virtual machines and run the tests in them and it's configurable how many workers will be on each worker host we run on fairly powerful machines so our main test boxes have got 30 workers so at any one time they can be running up to 30 tests that's how that works you are alright thank you so much thank you sorry I took a lot of time I think good job thank you for that introduction yes so welcome everyone and welcome to my talk on developing modern eBPF applications I'll spare the introduction like I'm doing networking at Red Hat mostly kernel and a lot of eBPF work recently I have a few notes up front I'm super happy to answer questions but please keep them until the end and we can have also lively discussion about that one thing like the second thing I designed to talk for an audience that is kind and knowledgeable and eBPF already so it helps if you have already written eBPF but hopefully you can follow along if you didn't so far the reason for that is that my goal of the talk is to show you new ways how you can improve your eBPF programs so you can switch from writing small toy programs to like writing normal eBPF applications like bigger ones where you can do more complex stuff like I said I'm working on networking mostly so my examples might be slightly screwed towards networking but that doesn't mean that they don't apply to tracing use cases it's just my examples usually come from networking alright so let's get started I want to cover basically four topics today the first thing is modern eBPF what do I mean when I talk about modern eBPF the second thing is I want to show you how you can compose your eBPF programs like the small programs you have into bigger applications there are different ways to do that and I want to show you three of them the third thing I want to show is testing so with growing complexity in the applications how do we test them how do we go ahead and make sure they do what they are supposed to do and the last one is I want to cover eBPF helpers and kernel functions so those are basically the APIs that you can use from your program and I want to show you how you can navigate that space more easily so you know what functions what APIs are there and which one so you can use so let's talk about modern eBPF what do I mean with that what do I talk about when I talk about modern eBPF I want to illustrate that with a few examples I have brought some code snippets here all of these code snippets are from the samples it's not important what they do right now but I want to illustrate a few things with them the first thing I want to illustrate is eBPF a while ago didn't have any loops you could not write loops that you are used to in any normal programming language was not possible in eBPF the only exception to that was you can ask your compiler to unroll the loop so basically copy the content like the body of your loop one after another and run that instead that of course increases the size of your code and it's not applicable to each and every loop it has to have an upper bound that you know at compile time already next thing that was not really possible was calling functions so you could not just call the function like you would in any other language it was just not there the only thing you could do is again inline all the functions so basically copy the contents of the function into your main program so you have a constant flow of execution or you could use so-called take calls which basically means at the end of your eBPF program you jump to another eBPF program and execute that instead but you will never get back to your old or previous function so it's not really a function call it's a way to combine programs but not really a way to call functions so let's see what improved in terms of loops eBPF nowadays can have loops like you can write loops in your eBPF program the first addition that we had there were bounded loops so basically loops that have a fixed upper bound as well so the verifier can really check does this loop end at some kind of time so you can be sure that it doesn't prevent a kernel from continuing with that it was not necessary anymore to unroll the loop your compiler might still do it but it's not necessary anymore and it covered some use cases but not all of them so we later on had further additions to the eBPF or the eBPF program environment that were the loop functions how I call them so basically the first one that I want to introduce is eBPF loop that's a helper function that you can use in your programs and what it does it takes basically a number of iterations and it runs your another function like a callback function that number of times and the number doesn't need to be fixed or constant or anything like that it's checked dynamically at run time so at run time you can pass it a number that you got I don't know from a network packet for example and run the loop isn't that many times that's like with that you can probably support all the use cases you have for loops you can do everything you want and then there are other functions that might simplify your life if you want to do something different so there is eBPF for each map element and like name already implies what it does it gives you a way to iterate over the contents of maps with keys and values so all the eBPF maps you can iterate over the contents basically it's also something that was not really doable before at least not easily and now you have like a very useful helper function to do that and like a more convenient way for that let's talk about function calls as well so eBPF now as well supports function calls something that was not possible before and if you now want to write eBPF functions you write just a normal function like you would a normal C code and the compiler translates it to real function calls no inlining anymore, no nothing your compiler as well can still inline it but it doesn't have to and you can really have function calls and with that you get all the benefits of calling functions you have better modularity you have like your code size goes down if you're calling if you beforehand had an inline function that you would call from many other places your code grows in size now if you have real function calls like it's reduced in size again and one nice thing about it is that every function is treated by the verifier as its own program so the verifier is that thing in the kernel that basically checks your programs and it checks each function as its own program that means that all the limits that the verifier has apply to one program or to one function at a time so for example the complexity limit it applies to your main program first and then it applies to the function you call second so it's you can write more complex applications because you don't run into the limit of the verifier that easily anymore if you break down your application into smaller functions so those things that I just said like the function call that's just a few examples for what improved in the eBPF territory so things that you can start using today now that we can split up our execution into like different functions let's talk about how we can compose all these functions back together into like big application or larger application I want to introduce three different or like two and a half different ideas for how you can do that and the first thing I want to talk about is how can I do like how can I compose programs together like how can I combine them at build time and right now we have a super simple linker in eBPF tool so that's something that was added a while ago you can now call eBPF tool and link together eBPF object files like you would have done with your normal C object files in the past it's really doesn't like the linker is not that complex like it is like a normal linker for C user space programs but you can really do the same basic thing with it you can have your code in one file you can declare your functions that you want to call in a header file you can write the code for that in a different file compile these things separately and later on link them together you can ship one big binary but you have your code split over different files so you can organize it nicely and so on that makes it like I said easier to structure your code into something that you can maintain more easily something that is very important if your application grows in size and of course with that you can build stuff like static libraries for example so you can for example have a team that maintains all the parsing functions networking focus parsing network packets is something that we do a lot and you need to do often and you basically the code is the same all the time because the network packets look the same all the time and you can go somewhere and build like you can build your library build your parsing functions into that and then we use them where needed like basically make an object file of that and link that into your application so that's a use case for like these linkers another thing is imagine you don't have all the object files available at build time, imagine you want to link stuff at load time or combine programs at load time the BPF tool functionality I just described is based on LibBPF LibBPF is that user space library that's there for basically all your BPF related wishes and it also exposes the linker functionality I just described to you as a program so you can programmatically link BPF objects at load time or just before you load the program into the kernel you can just link it together and then load the result into the kernel one thing that I could imagine you could build with it is you could for example have your application built already and allow users to provide you with like their code imagine you have like a big networking application and you have one place where you could offer the user or the users your program to collect statistics for example and they can provide you with an BPF object containing the program to generate the statistics and that they can extract from your program and you can link that object into your application at load time so the user can really make use of the BPF functionality still after you have attached your application so imagine something like a plugin system in easy terms right and the next thing I want to talk about is a bit more to be honest it's called every place so imagine you have a program that you have already attached that you've already loaded into the kernel and you want to change parts of that program so there's a functionality called every place it's available by assist called it's available from the BPF as well and it allows you to like swap out functions or sub-programs of your application that are already there so you can even if the thing is already attached at run time just replace a function of it and basically put in your new code it's a super powerful concept like you can do a lot of different things I've seen people doing interesting things with that but you have to be kind of careful because it's not it's not as usable as other features from the BPF the first or the main restriction is you cannot use it recursively so in your whole call stack like in your whole function stack you can only have one function that got every place at some point of time you cannot have multiple of them so as soon as you have used it once in your stack you should not use it somewhere else in the stack you can still replace that same function again and again but you cannot replace something up in the stack or lower in the stack and that combined with another feature so this functionality every place is often used in infrastructure like LibXDP for example uses that to attach multiple XDP programs to one network interface that's something that a kernel doesn't provide but LibXDP allows you to do that and that's based on the LibXDP on the every place functionality for example so if you are writing XDP programs you can probably not ever use every place because LibXDP already does that thing so every place is something you want to use in your infrastructure maybe if you're building infrastructure for the BPF ecosystem that's an interesting thing to use and to know about but it might be not that useful in your applications I wanted to show it anyways because it's super powerful and if you're reaching like the limits of what you can do with the other method I showed before maybe every place is the thing you need to really get forward so be sure it is there but when you're using it next up when we combine programs at runtime, load time, whatever so we're building more complex applications out of our programs we want to make sure that those applications do what they are supposed to do so let's jump to testing there is a way that probably a lot of you already know if you're using BPF or if you're developing an EBPF you're basically running your full application in some environment to test it so in networking it's usually a combination of network namespaces virtual ethernet pairs shell scripts bridges and so on and you basically set up a network like a virtual network within your computer have different namespaces and you run your application inside that those namespaces basically that's cool thing like I've written here it's used in self-tests it's used in demos I've seen it used for examples as well so it's a pretty simple thing actually right you can just set up your network and just run your application within that network that you set up difficult is how do you observe if your application is really doing the right thing so you cannot test your application directly but what you're doing in networking for example you're usually just sending traffic through application and see if it does the thing you expect and if I don't know the traffic is dropped in between do you know why it dropped you don't know it's just not there anymore and then you start looking where's the issue coming from is it coming from my application is it coming from if I run this same script on another system is everything in the same place that I expected to be all these different issues that you get if you're just basically running testing on your main machine so it gets hard to observe what is going wrong what goes wrong and it's also kind of brittle sometimes it's like you have a lot of race conditions there you set up network interfaces what does the system do with them and so on and another thing is that if you do stuff like that you need to change the system of your developers and I can say for myself I don't like people adding network interfaces to my systems I have like they're using IP spaces I might be using them as well and it's always causes some issues I want to show you one other technique that you can use to debug or to test EPF programs that doesn't require all these things and that's called test run or BPF test run so that's a way where you load your BPF objects just your BPF objects not the full application but really just the kernel parts or the BPF parts of it you load these objects like you would in your program as well into the kernel then you know already those programs pass the verifier and then you basically run the program without attaching it to the real thing so you don't attach it to your network interface or to your syscall but you just run it with a defined context that you give it like you pass a buffer to it with a context and you run the program with that so you can think of it like unit testing for your BPF program you take the BPF program give it a defined input and run it and observe the output that's of course super useful for network programs but the context is super simple it's usually just the network packet it can be used for most network program types it can be used for others as well at least for syscalls it can be used for tracepoints it can be used and the full list is in the link the slides are shared in the schedule so you can get the slides from there and click the link if you want to I want to quickly give you some hints on how to use it because the documentation as some parts of the BPF documentation are pretty sparse but my hint is take a look at struct BPF test run ops in the BPF for like to get started so you can really see all the options that you have for running programs in that testing environment and the core idea of it is you build a packet in like user space memory it's a memory you write your packet data and you do it craft the network packet as you would do it otherwise it's just basically you place your packet data there or your context if you're speaking about syscalls then you pass all these like the full buffer you pass the full buffer to the kernel to that BPF test run function and the kernel then continues to execute the BPF program in the kernel so it's not like running in user space running in the kernel on your buffer if you ask it to it can run the program repeatedly so that's interesting for example for benchmarking reasons if you want to run multiple times and after the execution the kernel returns the result of the execution so it returns for example the modified network packet if you're speaking about that it returns the usual return value of your program so you know what would have happened if that thing was like being run in the kernel at like a normal hook and for benchmarking reasons it returns the average run time as well so it's a pretty nice thing that you can use for unit testing like I said for benchmarking as well especially for networking where you have like programs that are usually time sensitive so you want to process the packets quickly so that's why you can use it for benchmarking as well so next I want to talk about the eBPF helpers and kernel functions those are basically all the APIs that you can use from your application it's basically a set of functions that you can call and for you as a developer it doesn't make a difference if you're calling a BPF helper or a kernel function or kfunk that's not different it looks the same it's just a function call the difference for you as a developer is that the BPF helpers are part of the stable user space API of the Linux kernel so there is a man page documenting them and that's part of the stable API so that will not change in a backward incompatible way so your programs using BPF helpers will always like if they run on a current kernel they will run on a new kernel as well on the other hand we have kfunks and they are not necessarily stable there are some life cycle guarantees attached to them they should follow some rules and those rules roughly boil down to we don't want to change them without reasonable justification and we don't want to remove them without reasonable justification and there should be an application period if we want to remove them but it's not guaranteed so if we see there is an issue with those functions and it's reasonably bad that issue then we can just remove the function or the kfunk if you want to use kfunks it's a bit more of testing you should put into place when you a new kernel version is released check if the function is still there and if they still work in the same way sometimes these helpers like these kfunks are also explicitly unstable so there are functions that are explicitly unstable one example for that is currently connection like contract access you can access the contact table from your bpf program and that thing is currently explicitly label is unstable so you well be careful when using it that's the main take away from that the list of helpers and the list of kernel functions is basically growing with every release so it's hard to like take a snapshot in time and say like this is what you can do because it changes like from release to release and then you can add new functions add a new functionality that you can call from your bpf programs and I want to quickly show you how you could navigate that space so how you could find your way around and how you can find out which functions you can call and which ones are available so for bpf helpers it's kind of easy so there is the data man page that's called bpf helpers and you can just type man bpf helpers and you could get a long list of functions and I invite you to just step through that once and see what's available the functions are somewhat descriptive like the names are somewhat descriptive so you can at least guess which area of functions are available I would not want to guarantee that everything is documented in that man page so there is this header file that I linked here or that I showed the path here you can go to that header file and take a look so all the functions must be in there bpf helpers must be in there that you can use for kernel functions or kfunc it's not that easy there is no man page documenting all of them so in the end it boils down to you need to look at the kernel source if you want to know which kernel functions or kfunc you can use some of them are documented in the normal kernel documentation but not all of them so a lot of them are basically only viewable if you look at the kernel source but one nice thing is all of these kfunc should be marked with underscore underscore bpf kfunc so if you just grab in the kernel source for that you should get a long list of functions that you can use and it should be almost complete there shouldn't be any other functions that you missed that way right so as a summary what I want to show you with that talk is that basically the kf development environment got a lot better in the last years a lot easier a lot more comfortable and so you can really build more complex programs more easily and one particular nice thing about everything that I show today is these features are not yeah these are not from like yesterday's kernel they are not super bleeding edge but they are present in kernel for a while already so if you have reasonably recent distribution or Linux distribution you can use these features today in your program like if you target something that is somewhat current you can use it today and you don't have to wait for the next two years for those functions to arrive and for those features to arrive and yeah with that I want to thank you for your attention and if you have any questions please go ahead so this is happening in a runtime run and previously you mentioned that now at the modern dvdf you don't get unbounded by the loops so you can have an unbounded loop so is it correct to say that at free play you can have now that if it's having a run time you won't need an upper run or lower run and you can have a number of loops so the question is basically if we can kind of escape with every place and the loops like the unbounded loops if we can escape kind of the limitation that it has to stop at some point of time no, not necessarily so everything still passes through the verifier and it's not really unbounded so the loop function for example that's also like it's a helper function as well so that's code that is not within your ppf program and you don't control it but the code that iteratively calls your callback function is part of the kernel so you pass a number of iterations to that function and then the kernel makes sure to call the next function a number of times and only that number of times and there is an upper bound to that so it's not unbounded in that way and no matter if you every place anything after that, that doesn't change anything yeah, go ahead so what I can add there it's not that they're not documented at all it's just not like the documentation is not exposed like you don't see like it's not in the normal kernel documentation it's not in the rendered online but if you search for k-functs like the ppf k-funct thing in the kernel then usually these functions have some kind of annotations on top of them saying like this function just isn't bad those are the parameters so it's not completely undocumented but it's not openly documented in a like online rendered way any other questions what exactly do you need okay, thank you so we were asking for one sentence basically what you can do now what was not possible for yeah or an example for that so for me the most interesting things are let's say these composability things so you can build more complex applications and really you're not bound to the strict rule I build my program first and then I load into the kernel anymore and then I change at runtime what is going on like in the most extreme case and what is possible for that is for example what I told before the libxdp so the library that's used for interaction with xdp programs they found a way to attach multiple xdp programs to one interface using every place and beforehand there was a one-to-one mapping like your network interface could have had a program you could run it but that's it and they made it possible that multiple programs can attach different programs to the same interface that's for example something that is enabled by these new functionalities alright, thank you very much so I'm going to give it like two more minutes am I already unmuted do you know how I could point with this yes there's a leader aha, great the red one great, thank you sure so so, hi everyone I'm here to talk about glibc I've tried to put this together as a sort of call to action for a first time contributor or a fairly new contributor but I'm also hoping that maybe it's useful at least entertaining to people who are experienced programmers and already know their stuff who am I, my name is Arjun I'm an upstream glibc contributor I also co-maintain glibc in Fedora and Red Hat Enterprise Inux which of course means that I work at Red Hat so I'm going to try to this is the last talk I want to make this really quick to talk about whatever I have to say and leave some extra time for questions and answers I hope I make it it's really my first talk so I'm not sure how I'll get the timing right but I'll try my best to leave as much time as I can for questions and for people who want to maybe leave or leave to party right so I'm going to go with an introduction I talked about seven steps in my title so I'll quickly jump into the seven steps and then I'm going to walk through a patch a glibc patch, a recent glibc patch just to kind of show what kind of goes into writing a patch and then I'm going to talk about what you could do to contribute to and help with glibc and then of course questions at the end and we're off so I was first introduced to the C language in high school I knew a little bit about stdio.edge and malloc and to be honest it was mostly magical incantations that I wrote in the middle of my program to print and to get input and to allocate some memory I really believed it was something that's happening within the compiler and kind of didn't really think too much about it but very soon I realized for example is a function that can be written in C so it's not really magic there's something going on there obviously eventually I got into this field and I know a little bit more about all of this well not all but some of it it's a pretty white topic so I know that a lot of it is actually mostly written in C mostly written in C so what is glibc it's the standard C library we have all of those functions and a lot in between for example one might think that main is the first thing that starts executing when you start a program but there's quite a bit that goes on before that that is the loading of the program placing everything in the right places fixing up function addresses and then finally you call main and then you go on with the execution so there's a lot more to glibc than just providing the functions it is the runtime okay so next we're going to talk about why it's maybe useful or why you might want to contribute to glibc I think the first reason at least for me is that it's very high impact there are millions of installations it is the C library for the majority of non-android linux based operating systems if you make a change in malloc for example that shows up on the critical path you're looking at trillions of executions of the code that you wrote in a week or something like that I don't know the exact numbers but you can think about every time malloc goes through your code's in there doing something so that's what actually makes me very happy about being able to work on this stuff the other thing is we actually have a fairly actively developed piece of software it's not really like some arcane old thing that's never really updated we have over a thousand commits a year which means we regularly add bugs that need to be removed my personal experience with the community is that it has been very welcoming to me very kind mistakes are welcome and accepted and understood and commit bugs once in a while and help each other out to fix it it's like any other piece of software really we recently I think recently is not really accurate but I think it's been more than a year already we have weekly public video patch review meetings so people who have recently submitted a patch can actually show up to that meeting there's a link somewhere in our wiki show up to the meeting and talk about your patch or say that you want review and someone will be assigned to look at it a lot of the regular attendees are basically like regular contributors I don't show up very often to that meeting for example but once in a while I do and try to find or get assigned a like maybe a beginner's patch because I do care about this I do care about it makes me happy to it's first of all easy to review a patch by a beginner and it makes me happy to see that we have another person who made a contribution and who might make more contributions it's got this kind of multiplying effect so we have all of this and we do look out for patches from new contributors and we also have a code of conduct that is a work in progress so yeah we do care about being welcoming be kind to everyone and trying to get as much as possible contributions from people who are willing I said seven steps it was a clickbait to be honest I didn't know how many steps it would be I don't think there's a fixed number of steps but I did shoehorn the seven steps here you can see them so you do a git checkout and this is a bit of an idiosyncrasy you need to be building in a separate directory you can't really build in the same directory as the source tree I don't know the reasons for it it's something to do with the build system I don't know most things about glipc to be honest so you're building a separate directory and you need to make sure at configure time that you provide a prefix which is about where it's going to be installed it's just a couple of idiosyncrasies to building glipc that don't exist in a lot of similarly packaged applications so let's say that you're trying to fix a bug a good place to start is by maybe adding a new test that fails without this bug having been fixed so that's a potential step three, you implement the fix you do some testing we recently added a couple of recent is always a bit of a longer time span when it comes but maybe I think a couple of years ago maybe slightly more we added a couple of scripts to help you run a program not on the system's installed cLibrary but on the one that you just built so there's test run for that and then there's you know you need to do a lot of special things to get gdb to pick up the intree freshly built glipc sources sorry not sources at this point of course executable and then run it with a test program and so we have the script to help with that as well and you can use that to kind of check out how your test and your fix is working and then eventually you delete the build directory, you reconfigure you run make check, you make sure everything's working fine and perhaps you submit a patch to libcalpha at sourceware.org that is our mailing list we submit patches there, we discuss patches there right is that the end of the talk maybe not so I will now go into the anatomy of a relatively simple patch, I try to look for something that is on the order of magnitude halfway between a fixing of a typo in a comment to like a entirely new let's say feature that changes dozens of files, somewhere in the middle of that is this, it's a fairly simple fix that went into gilipsey recently, Mike who is an experienced developer and a prolific in the gilipsey sorcery, he submitted this a couple of days ago, I reviewed it for him and it's basically a fix to a function called stir error and apparently this function must not return null we'll go into why so for some reason stir error the function was returning null in some cases and it's not allowed to do so and then Florian goes on to explain that we made a recent change where stir error was implemented in terms of another function, stir error underscore l and that's what caused this sort of regression that needs to be fixed what is stir error and stir error underscore l so stir error is basically if you know the error number, it takes in a number which potentially is the error number and it returns a string corresponding to that number describing what that error might mean obviously if you put zero you get success and I did not know this and I really don't know if this was a joke or if it's I don't know, divine intervention but stir error 42 returned no message of desired type, I have not read the book but 42 is apparently an interesting number and I thought it was funny that this is the reply that you get when you try to find out what 42 means so stir error underscore l is a very similar function that returns a string in the current local which might have a different language than the one the in a given local, sorry, not the current local the current local is the one in which the program is running so the program is running in local x you want the error message in local y for whatever reason you use this other function obviously you can now see why stir error can be implemented in terms of stir error underscore l you just pass the current local and you get it back so someone made that change which was definitely an improvement we don't want to duplicate code but it caused a regression and why was it a regression? stir error is not allowed to return null but stir error underscore l actually is allowed to return null for some reason and the details of that are actually in this little bit in the POSIX documentation for these functions POSIX tells you what these functions are allowed to do in what circumstances and basically the POSIX manual says that whether successful or not stir error must return a pointer to a generated message string but stir error underscore l does only need to return it upon a successful completion or fail it can return null