We got a couple more people coming in, all right. My name is Justin Forbes, I'm the Fedora Kernel maintainer and have been for a disturbingly long time. What I'm going to go through today is, this is the first time we've done the State of the Fedora Kernel talk since NEST 2020, I believe. So there's been a whole lot of change and anybody who's working with the Kernel package would, well, it's ugly. It's ugly for a reason though. So what I'm going to go into in this talk is a little bit on the end user experience, stable kernels versus raw hide kernels, a little bit there. I'll go into a little bit of the Kernel contributor developer experience. And then I'm going to go into people who are debugging testing patches, things like that upstream and how you deal with that. From a Kernel perspective now, if you look at the spec, it's messy. It's not as clear how you do some things. So I'll go through that. I'm going to try this all in 25 minutes. So instead of having a nice flow, it's going to be a little bit disjointed, section to section. So the first thing is stable Fedora kernels.  And why are we getting so many kernel updates? One of them is Fedora does try to stay very close to upstream. We push almost every single stable release as an update. If you want to know why we do that, there have been 117 kernel CVEs so far this year. Now, a lot of those CVEs don't affect everybody. Some of them only affect very specific hardware. Some of them affect older kernels. We've already fixed them, but just in 2023 alone, there have been 117 CVEs so far. One of the other things that happens is we get regressions as these updates go through and it's unfortunate. So we got to get bug fixes and try to get those regressions through. One thing I did want to stress here with karma is you are going to see sometimes, oftentimes we will push a kernel that I know breaks users, breaks specific users, right? If somebody says I have problems with my AMD graphics card and it's a specific card or something, I'm not going to hold back fixes for a bunch of other users just because of this one specific problem, this new regression that's come in. The idea is because we released so many kernels so quickly, hopefully that regression will be fixed soon as well. Now, there are exceptions. There was an XFS issue that could corrupt data. We did not push that kernel. We held on to it until we could get that issue fixed and get it pushed. Data corruption, things like that are not things that I'm willing to let go as a regression. But if a specific piece of hardware doesn't work or sounds not working right or something like that, a lot of times other users still need the fixes in those kernels. Another thing about the stable Fedora kernels is if you look at the rail kernel or the rawhide kernel, you'll see there are actually a few extra patches in that. There are some rail specific things and the reason those exist is because of ELN. We build kernels with most things in rawhide. It gets rebuilt for ELN and it's the same package. With the kernel, it's the same disk it, but the ELN kernel build is essentially a rail. It is exactly a rail kernel. It's got a rail config and it does have a couple of patches that we don't have. One of the things that I do as I make sure that when these patches go into rawhide, is anything that's rail specific, anything that is not wanted or needed for a Fedora, Fedora has to have if-defs so that that code path is not executed by Fedora users. There is a config rail differences config item that basically says we're just going to bypass all of these chunks of code that rail is added. Rawhide that's there. Running the kernel should be no different than it would be if those patches weren't there, but it does make it a little bit more difficult if you're dealing with upstream, say I've got a bug and I'm talking to upstream and they want you to apply a patch and the patch doesn't apply cleanly because there's this weird rail code there. So stable Fedora releases, those patches are removed. And part of the reason why is to make sure we stay close to upstream. So the other thing about karma is if you get a, whoops, if you've reported a bug already, if a bug say happened in 6.4.3 and that bug is not fixed in 6.4.4, negative karma is really not helpful there because what that's saying is my bug is more important than any other bugs that might be fixed by this. I'd need the negative karma if it's a new regression because I read all of those. It's good to know we've got a new regression, what that regression is, but for kernels in a row that you're applying negative karma to for a regression that happened five kernels ago isn't necessarily helpful in that regard. So that's kind of the end user experience part. And now we're going to talk a little bit about kernel maintenance. And the reason that kernel maintenance is important is if you want to do your own kernels, if you want to look at our code, if you want to do any of those things, it's much easier. What we're doing now is everything is maintained in a source git repository on GitLab. And that's the kernel arc repository. In that repository we have branches, there's an OS build branch which is the raw hide branch. And then we have a branch for each stable Fedora release. And when I say stable Fedora release, I don't mean Fedora 37, 38. I mean Fedora-6.4 is the 6.4 kernel series. And the kernel is the same for all versions of Fedora that support it. So 37 and 38 are both on the same 6.4 series right now. And it's actually one script that just updates the disk git for both of them when we do a stable build. That sometimes changes when you have a release going into life, like I didn't rebase Fedora 36 when we had less than a month left. So it stayed on the previous version. In that case, that doesn't mean we won't do updates. It just means I'm not going to do an update unless it's a security fix, something like that of some importance. It is end of life from a new code standpoint at that point. Another thing that's interesting about this, though, maintaining everything in source git, it's an exploded kernel tree. And we have scripts that actually generate the disk git, and they do it by creating a source RPM and then just exploding it over the disk git and adding everything again. That means anything that's done in disk git would be overwritten. Now, that's a weird workflow and I know a lot of people aren't used to it. So, you know, luckily we don't have people frequently committing to disk git in kernel. But we do, I do look at the diff every time. So when I commit, I actually have a script that does a commit and does a diff. So I'm looking at the diff before I push, and hopefully if anybody did change anything in disk git, it would be noticed and it could be picked up and brought into source git correctly. It also means, though, kernel doesn't really do pull requests from disk git on pager. We do have on peer. We do have merge request on git lab. We highly encourage them. Several people in the community have contributed and it's been a great experience, at least from my end, and hopefully once people have figured out the process, it's good on their end, too. One of the downsides to doing this this way, well, when you're looking at the kernel spec in disk git now, it's gotten pretty nasty. And I hate to say it, but it's going to get a lot nastier. And the reason that is is because we've introduced variations. There are variations that are there for RHEL now. There are variations that are coming for Fedora. So the real-time kernel is supposed to at some point be merged upstream. I'll say any day now. Clark would really appreciate that, but we don't know when it's going to be. It's been any day now for a year or two. When that happens, there's a good chance we're going to do a Fedora real-time kernel. There's a good chance that we will do a 16k kernel for ARM devices. Those things are all variations. And while it's very easy to manage them from the build side, it gets pretty ugly in the spec. Eventually, I kind of hope to break those things up and make it to where you don't edit the spec or the spec template directly at all. You're just editing chunks which relate to what you need to do to make it all a little bit more manageable. But we're not there yet. Now, the configs portion of all this. If you need to change a configuration item in your kernel, say you want to turn on a driver that we haven't turned on or you need to turn off something, any time you need to change the configs, they're terrifying. There's thousands and thousands of files in the Red Hat directory. There's a Red Hat configs directory. And inside that, you'll see Fedora directory, a common directory, and a RHEL directory. The idea is anything that is unique to Fedora is set in the Fedora directory, and RHEL might be set differently. So anything that would be unique to RHEL is set in the RHEL directory, but anything where they're set the same on both would end up in the common directory. Tries to make it a little bit more manageable, but it's still pretty scary. And it gets you a little bit more interesting when you say, I want to turn on the config option and you find no file there. If there's no file there, it means that adding one and turning it on will do nothing. The reason that is is because that when you build the kernel or when you actually, we have a script called make disconfig check even as well, that you don't have to build the kernel to even get to, it goes through and builds your configs and checks them and make sure that they're valid. If something is unset, it's going to say, hey, there's nothing set here and you have to set it in one way. Now, the reason that you don't have a file there, if something is unset and you're trying to turn something on, you know that's a valid config item, is because there's a dependency that it's missing somewhere. So if I don't turn on Netbender Qualcomm, for instance, there's four drivers underneath there that it doesn't even ask about, it's not going to check because I haven't turned on that dependency. So if you are changing things, and this is important because we even have internal kernel developers that stumble on this, it's just, unless you're dealing with Kconfig frequently, it gets somewhat confusing. If there's not a file there, it means you need to find out what that Kconfig option depends on and perhaps go change that as well. Once you do set your configs to what you want to do, there is a command make disk configs check which builds all configs for all arches and checks them to make sure they're valid. It will tell you if something is missing, sorry, it'll tell you if something is missing, it hasn't been turned on or hasn't been set in either way. It will also tell you if there's a mismatch. So let's say I set a config item as a module and for some reason, due to dependencies, it's forcing it to be built in and can't be a module. It'll say, hey, you set this as module, the generated configs are showing it as built in. You need to go fix that and it won't let you continue without that. Please, please, if you're doing a merge request for the kernel changing any config items, run this command first because if you don't see I will fail, it's going to have to change anyway. The other thing I'm going to say is help is on the way.  Others inside Red Hat know how bad this is and there have been several discussions and there's even been, there's code being generated now by someone within Red Hat that will be made, that will be made public as part of this kernel repository where you will be able to manage configs in a much easier way. You'll have a way to say, I want to turn on this driver and it's going to come back and say, okay, as a result we also have to turn on these things. Is this okay? And that's it. It'll be a lot easier to manage but it's going to take a little bit of time. So I'm hoping by the end of the year, but we'll see. In the last bit, historically people dealing with Fedora kernels, if they get a patch from upstream, they want to test that patch. It used to be a lot clearer how you add patches within the spec file. It's still not horribly difficult. There are two ways that you can really do things. If you're working in disk it, you can actually just add that patch to a branch and make dist srpm and it's going to generate a source rpm for you to test. But if you really want to work within disk it or you're working from a source rpm itself, we have a blank patch called Linux kernel test patch. And you can take a patch or a series of patches and cat that to that file and it's already there. It's already applied so that you don't have to go through and mess with things that way. We also have an option in disk it for config overrides. So if you want to change something in your branch, if you're running a branch somewhere, you want to change a config. You don't want to mess with our configs about conflicts as you update over time. There's a directory called custom overrides. Red Hat configs custom overrides. Follows the same arch layout. When all configs are generated, the last thing that happens is we go through and say anything that's got an override, we apply that last. So the order of operations is common, fedora, and then custom overrides. And so whatever you set there will be the final thing set. Again, you still have the problem of worrying about dependencies, worrying about mismatches, those things. But at least you can maintain your branch with custom overrides and not have to worry about updating and us changing something. The merge is not working correctly. That one quicker than I thought. Sorry, I tried to rush through some of that and I'm happy to discuss anything you would like to discuss as far as questions on any of the topics or anyone? Yes. You said that in raw hide, the rel patches are applied. So if we're using raw hide, we're kind of using the rel kernel? No. Raw hide disk git has rel patches applied. But like I said, anything specific there, so like there's some code to support a different type of multi-pathing that rel used to support. Anyway, upstream has rejected that, so rel patch, they are caring for compatibility reasons or whichever, but that code that enables all of that is if deft in the kernel itself and it's under config rel differences. So rel and thus ELN has config rel differences turned on, meaning that code path exists. But raw hide does not. So functionally, the kernel is the same as it would have been on a stable Fedora branch, but the code is there. Anyone else? I looked at this whole workflow a few times in the past and it always struck me as very, very complex. I mean, I know that there are reasons for this, but like long-term, do you think that there's chances that it will become simpler or do we really need this complexity? Some of the complexity I think is required and some of it we can simplify. Like I said, the config stuff I know is a nightmare. We're trying to simplify that. We're trying to break out the spec in ways that it's easier for you to do a custom variation, do custom kernels, things of that nature. But as far as the doing a merge request through GitLab, that whole workflow, it is kind of I think what it has to be right now. Now part of that is rel requires controls on everything.  And what you'll find out is like if you submit a merge request to Fedora 64 or like any of the stable Fedora branches, all that it doesn't go through any of that process. If you do a code change in raw hide, it does have to go through those processes because it requires acts. And then if you do config changes, as long as it only touches Fedora, again, it bypasses all that process. Now, I also have a bypass lever because I don't actually build kernels in raw hide out of OS build. I build it out of a directory called arc latest. Excuse me. Arc latest is regenerated every night. And what it is is the contents of OS build plus any patches that are not or any merge requests that are not merged, but they're listed as include in release. And the reason that's there is so that I can make sure that Fedora is doing what's best for Fedora without having to wait for rel acts or rel process for those bits. I still have the ability to react and do everything that I need to do and what's best for the Fedora community. Now, everybody on the teams that I work with internally are certainly most of them are actually they're all running Fedora. So they're happy to they think about Fedora, they're happy to have things going for Fedora. But at the end, they're worried about getting code in and doing things for rel in that rel process. My only job is the Fedora kernel. And so that's, you know, I'm the the lookout to make sure, hey, we can't overstep bouncer, we have to make sure that we have ways around all of this to do what's best for Fedora at all times. Is it on? Yes. When can we expect your new book then in the art of kernel maintenance? I don't know that that would be quite the book, but let's say when I started kernel maintenance I had a lot of hair. About upstream and not upstream bug reports. So what is useful to have in Baxilla report and what is not useful assume you have a wireless car that not quite works as an access point. Right. Is it useful to have the Baxilla report there about it? I assume it would be in upstream but I don't really know. So we try to stick as close as possible to upstream. And yes, most of the time it is going to be an upstream issue. Now it's good for us to know that there's several people with these types of issues because one of the things I have to do is watch for patches upstream. So I don't want to bring in patches to the Fedora kernel that have been rejected upstream or have not gotten any comments or anything else. But I am happy to apply patches that fix bugs before they've made it to a stable release upstream. So I'll backport patches from Linux's tree to a stable release or even from Linux next to a stable release to make sure there's a lot of users. Yes, ideally if you can deal with upstream directly it's going to make things better but I still need to know that it's happening. Okay. Anyone else? I've recently started using Rawhide more and I've noticed when I do a system upgrade there will be what looks like two different versions of a kernel or multiple in a day. Just for an example it was 5.0 RC3 and then a long number afterward and then later on in the day I was just checking for other updates and it was the same 6.5 0 RC3 but another long number could you explain that? So the long numbers are the get, it's the get commit ID that that kernel is based on from Linux's tree. Doesn't matter what we've changed on our side this is Linux's tree merged in. The reason you might see two Rawhide kernel builds in a day is it just depends on what time of day the kernel was built. I try to do one every single day but it might be the first thing I do in the morning or it might be something I do later in the afternoon so if you update in the morning you might get yesterday's kernel and then you update in the afternoon you might get another kernel that way but usually it's just one a day. The only time that I would ever do a second is if there's something that's a major major security update or like I said if there's reports of data corruption or something like that that are serious then I might if we've got a fix for that we might push that quickly but usually it's just one a day. It's usually also built after the build usually it'll land after about 9 a.m. central time in the U.S. so well the right right the the other thing we have a script that runs every day that merges in Linux's tree to OS build and that script runs at 4.30 a.m. is my time and then then script an hour after that that builds the ArcLatest directory so even if I'm traveling the scripts run when they do I can manually trigger them if I need to but I usually don't I just wait till they run. Hello I work for the QA team and when I look at kernels at Bowdy they usually updates the stink usually they get a lot of karma quickly a lot of feedback do you have any like approach that you take how long you keep the new kernel in updates testing before you push it? Yes and no so one of the things we do we turn off auto karma we don't want the auto push we realize years and years ago that we would you just skip updates testing we usually get enough karma to push immediately so the typical wait now because you're doing one to two kernels a week is it sits in updates testing no less than one full day and we'll push after that but a lot of times we just kind of wait until the next kernel comes out and we push to stable and then that one will go to updates testing obviously if there's a critical security bug or anything like that then we push it through as quickly as we can sometimes even with Kevin and QA helping out we end up with heroics and people going above and beyond to make sure that we can get a serious security bug tested push to testing given karma and push to stable within number of hours. I'm asking because sometimes it happens and I think it happened recently that some part of our user base is affected for example certain AMD cards no longer boot and usually you can see the negative feedback only after for example two days three days because it doesn't affect everyone so it will take time until the affected people update realize report the feedback so if for example new releases you wait for at least three days or like what's your approach to minimize the impact that so that was an interesting one I didn't want to push that kernel when I did the idea is yes you give it at least a day in updates testing and usually you'll get feedback by then AMD issues originally I thought it affected fewer cards than it did it turned out it impacted more cards and so there was a second kernel that was never pushed and then we had another update that came in that fixed some of the cards but left some of them still broken unfortunately there was also a critical security update there and I mentioned that in the body when I pushed it we had to push it because of this I'm sorry we'll try to get this regression fix as quickly as we can okay and I do have one related question extra so if I see some people complaining on discussion or somewhere that they have a hardware related problem and it seems to be like really specific it's not like tens of people complaining about the same thing what's the best advice that I can give them like report back in bugzilla or reach the upstream if possible but it's much harder to talk to the upstream especially kernel it's somewhat easier with Mesa and stuff because they have GitLab and everything but for kernel and mailing lists and everything it's very difficult for regular contributors so what's the best approach to do Red Hat Bugzilla is probably the best approach there so we wish you could work with upstream but I understand it's difficult for me even to remember this upstream used like for great you want to file the upstream bugzilla he'll use it but you go to another sub system there's a kernel bugzilla yeah you can enter in a bug for it but they'll never look at it but every sub system has a different way of working so that's kind of one of the ways we have to help I'm out of time out of time you can ask the question I'll be around all week and happy to answer any questions or take any contributions do we have Miro Slav here already Miro Slav can you please all rise thank you hello thank you everybody so he's going to speak about the case of SBTX versus Fedora thank you it's usually habit that you sit down when the jury say you may be seated but you are forgiven so members of the jury your duty today will be to determine whether the defendant is guilty or not guilty based only on the facts and the evidence provided in the case and we will be talking about the case of SBTX versus Fedora so thank you for the coming and thank you even the audience will be able to see what's going on in the case of SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX or SBTX yeah he made the system out of the blue because in that time what was the time Tom? 2003 yeah and there was no standard and probably should speak on the mic yeah and in that time there was no standard about licensing at all so when the license was GPL version 2 yeah Tom said GPL v2 string for that, and we live with that for a long, long time. Much time later, we had some problems with the software. That's when the stuff and Sbom started. To standard right now, we have SPDX and Cyclone DX, which is used for the Sbom, and one of the things for that is the licensing. So licensing starts to be standardized, and SPDX used the identifier for separate licenses. So when you use the SPDX identifier, you know exactly what license it is, and it's never a whole family of the licenses like we had in the past in the Fedora. By the way, who knows what Sbom is, and who doesn't know what Sbom is? Okay, so we have some people in the audience. I may refer you to my presentation on devconf.cz this summer where I spoke about what the hell is Sbom, but it basically is something like that, this picture, and it describes your system. Because usually the requirements for the system is just the top level, and you never know what's below. So it's the map. Some people like the metamorphosis list of ingredients, and you know where on the end of the list is the chili. It can spoil the things. Yeah, there's a question. I prefer saying your honor. So in this picture, the fun is of this teeny place, but from the Sbom things, we actually don't thinkers. Sbom doesn't care about sustainability of the things, whether it's sustainable in the future or not. You just describe the system and to note that, for example, this block has some security vulnerability. It just provides you map of the system and know what exactly there is. A lot of things for the Sbom, you can fetch using rpm-query all, and you get all this information. That's like name of the package, upstream website, what files it includes, et cetera. But one thing I think is the licensing, so that's part of the Sbom. And that's why we are doing the change in the Fedora itself, to move to something which is standardized, because every software right now in the world, which do something with the licensing, use some standard, and for what I most see is the SPDX. So if we want to continue further and be first, we need to use the SPDX. So last year, we started Initiative and a Change Proposal to Move to SPDX Identifiers. This is where we are right now, and the blue part is which text we already converted, which is pretty awesome. We are at 40% right now, which is a good number. It's starting slowing down a little bit, but we are doing a lot of stuff. We are adding like five new licenses to SPDX every week or something like that. So it's pretty fast for a human, slow from the whole distribution point of view. And the yellowish and reddish part is the part which we have to finish yet. And the yellow part, I mark it as trivial conversion, which a lot of people hate me for that, because that's trivial from technical point of view. But when you start speaking to lawyers, it doesn't seem to be as such trivial as it seems to be. One of the reasons is that we are changing how we evaluate license. Previously, you can evaluate the license, and now we don't evaluate. So some systems say, projects say, this is GPL version 2, or if you are shipping with Fedora, it's MIT. And previously, you may choose MIT because that was for Fedora. Now you should say GPL or MIT, and it depends on the context. So right now we have to evaluate all the licenses as well. So the trivial part is not so trivial, and the red part is not trivial at all, because we usually don't know to which part it should convert. And the non-trivial part looks like something like this. So if you try to convert MIT and BSD license string, we give you a hint that it's one of the MIT choice and BSD choice. And you actually evaluate the license because previously in the Callaway system, BSD, MIT and other licenses referred to whole family of licenses, which is right now, not possible. And now I want to meet your defendant, and it's a form of quiz, and you will be the jury and decide if it was the right answer. And I want to ask you to all stand up, please. And here we have the question, and this is one of the change-owners. And who is this guy? And if it is David Kanshark, raise your right hand, and if it is David Kanshark, raise your left hand. And I will shortly reveal the correct answer, and if it is incorrect, please sit down. If you guess it correctly, you can stand down. Credit for this quiz go to Radek Vokal, who introduced it in DEF CON. So guess who is it? David Kanshark or David Cantrell? Raise your hand. The change-owners. This is this guy, this guy on the board. It's not so visible. And it's David Cantrell, and he's here in the... Not on the board. And second change-owner, is it Emily Lovejoy or Gillay Lovejoy? This one will be tricky. One, two, three. It's Gillay Lovejoy. And who is this? Is it Miroslav Sukhi or Marek Sukhi? It's Miroslav Sukhi and it's me. And now slightly different. Who is Richard Fontana? Is it the guy on the left side or the guy on the right side? Both are Richard Fontana. This is Richard Fontana, the lawyer, and this is Richard Fontana, the actor. It's easily like Richard Fontana, the lawyer, is working on the change proposal. But this change was tricky and both questions are correct. And when the migration to SPDX in the whole Fedora will be finished based on current approximation and estimation, will it be sooner than October 2024 or later than October next year? And this is hard. And it varies because the estimation started to be summer 2024 and you see the graph was slowing down. So right now the estimation is December 2024. So you can do better if you migrate your packages. And so some people are still standing. So last question. Oh, sorry. Do you like the migration to SPDX identifiers? Yes! The correct answer is yes. And yeah, I was looking actually for standing ovation but you didn't get so many questions, so you are not standing anymore. So my fault picking doing that. So yeah, so that's basically all of I presented because I can speak about it a long, long time and I reserve more time for your questions. So what's your question? And then there we have. It's not so... I will rephrase your question. Okay, thank you very much. So my question is this change only for the licenses or you will publish the SPDX files with or the requirements provided and things they get to the Fedora in the format that will be accessible to the scanners or security scanners and things they get. Uh-huh. Because sorry, tomorrow I have a little bit of a supply chain in Rail and actually Redcat has already started to provide in Beta in one of the portals so I'm curious how it looks in Fedora. So again, the question is like... If Fedora provides the SPDX but not the license thing, the packages but the SPDX has the files for the individual. So as far as I know, Fedora provides SBOMs for the containers made of the Fedora but not for the whole distribution and even the SBOMs for the containers are very easy and not going down into the details and just for the others, like the SBOMs can be really, really huge, it's a huge rabbit hole because you can have SBOMs for the artifacts that's what we are actually producing. Then you have SBOMs which is for building systems which includes not just the map you've seen but other maps which is required for the building, the packages as built requires. You have the SBOM which includes stuff for designing so other stuff which you need to for design it before you start building that. For example, Enscape for drafting the logos and the other way around you can have SBOMs for deployment because your system may need Postgres DB but it can reside on the different system so it doesn't need to be tracked in some scenarios but in some scenarios it needs to be tracked or S3 which is provided by AttaVendor somewhere on the internet so when you say providing the SBOM it's a really wide area but for our change proposal I don't dive into that because that's a really huge topic and right now I'm and we the change owners are focusing just on the changing the license identifiers even that is a huge topic and what we are and even in this area there was a proposal that we do the analysis on every commit to this grid which is a really great idea and one day it will likely happen and there are some tools that actually do that for example, SUSE has the cable system which we probably plan to use or investigate but if we do that right now we would have to work and build the system for both the Callaway system and the SPDX systems and somehow make it live together which will be tricky so I pursued others that we first make great everything to SPDX and then some two years later maybe is my expectation we will do some system make some system which will warn you every time you do change that hey there may be new license or some license disappear so this is the future in the future Federa probably I have a second question but I don't want to I will give chance there and then maybe back to you My question is a bit twofold I was wondering if you have any view on what the most common licenses are in Fedora right now since you are going through all the changes and doing the statistical analysis and I was part of that What is your view on the proposal that has appeared recently to follow suit what Debian does and just sim link to copies of common license tax instead of having multiple copies of the same license So the question is common licenses I don't dare to say each license is common in Fedora We have a lot of MIT, a lot of GPL version 2 or later, a lot of BSD I've seen some pretty long list licenses like 800 characters long license identifier Right now because we are in the middle and we mix the call away system as PDX and the call away system itself it was not defined in any form but one wiki page So later when we finish this work it may be actually easy to determine the whole license for all Fedora distribution because there is a library which evaluates the licenses so we may put all the strings beside and concatenate with all operators and it evaluates the license for all Fedora distribution so that would be awesome and the idea about linking to common licenses actually last week Richard kicked this off on the Fedora legal list So it is proposal it has some problems like what is common license, what is not so for others like GPL version 2 may be common license and you may link to that but there are some other licenses which has the credentials contribution like at the top copyrighted 2024-3 by Miroslav Suki and it's part of the license and license checkers ignores this contribution thing but the license itself says it's free if you include this license in exactly this form in the software so you can't actually link it to some common example where here comes contribution because there has to be that contribution Yeah, yeah, yeah that's the part where things start complicated and whether it's sufficient to link the header or whether you have to include the part it's tricky and right now we have like 20,000 packages to go and we still defer some stuff like copyright only or CC0 already debatable stuff so we just noted and we will still have to do some fixes later in some cases so there is a lot of stuff and not sure whether like saving one kilo of text is right now the biggest priority but yeah, people are trying to address that and it's fine and I welcome such an issue at it Okay, my question is also because there are multiple formats for SBOM and why you choose one over the other for example as you have the second case and why you choose this one, SPDX instead of Yeah, so the SPDX actually was not SBOM's thing at the start it was the license management project so at the start they have huge list of licenses and as far as I know they are still the only one which actually care about the license is too much the Cycon DX is the new one and it's focused on the tracking vulnerabilities so they don't care too much about the licenses and they care about the vulnerabilities in the blocks so I'm not even aware whether they have some list of the licenses they don't defer it so much like the SPDX so my wild guess and I may be incorrect is that in few years even the Cycon DX will use the SPDX list because it's really comprehensive and very well maintained So can I ask Mike? Okay, your honor I was investigating one of my victims and there was a problem in evidence and in layman terms, is there some tooling that can help me with converting my packages to the SPDX valid license? Yes, thank you for the question and it's in the package license validate and if you're on license of Dash Fedora to SPDX with the old system it will tell you to which string you can convert it but there's a problem with license evaluation so you may run the checker and we have license check, scan tools a scalone and every tool has its own problems so there's no thing which I can point this is the best tool you can use and we are still evolving and the ecosystem is evolving Your honor, I just want to add to that one annoying feature of that tool is if you already have an SPDX license it tells you it's not a valid license string I may address it after this presentation because we are already at the end I'm aware of that so we are at the end of the hearing thank you for the coming Thank you very much Thanks a lot, that was fantastic The next we have Tim Flink No Sorry You are the one, right? I'm sorry if I... Don't worry, I was called worse I'm sorry Thank you very much I might just... Fuck it, I'm after putting off the whole thing Sorry Thanks Yeah No Hello everyone We are starting our next talk Please settle down at the back Okay, so please welcome our next speakers Effie Maloney Ifa Sorry I tried Okay, I'm brilliant at that one So the topic is State of Community Applications and Infrastructure Thank you everybody, all the best Clapping before you begin, excellent So I'm not Effie, I'm Ifa But it's close enough, people have called me worse Welcome to the talk on the... Actually Question, because we like quizzes here So can anybody tell me what team takes care of the community apps services and infrastructure for the centres and for our projects and if you're a member of that team you are not allowed to answer I'm not a member of that team Yeah, Nick got in there first Go first You get a prize Yeah Prizes will be thrown Yeah, so Community Platform Engineering Team This is a discussion just around the overview of the team, what we do and Akash is going to go into a bit more detail then on the applications and services that the team actually supports and maintains, etc. So, like I said about us, we are Red Hat's sponsor team We are all paid by Red Hat to take care of the Fedora and Sendos communities and it is a very nice team to be in because everybody who's in that team really does care about those projects and it's nice that we get paid for that We cover a lot of ground on the team We have operations, day to day stuff We do some development into apps and services that may either fall into a little disrepair and we also then maintain lots of apps and services Like I said, we cover a lot of ground We have a couple of little teams in a bigger team that we all make up together We have the Fedora infrastructure team We have a Sentos infrastructure team of one Fabian We have our release engineering team and then we have the initiatives team that spring up if we decide to action any pieces of development work We have some cool stats to share with you as well today and the presentation put together by Akash is all very cool laid out with lots of data and metrics and things So this is the first in-person flock since 2019 It has been a long time since everybody came together for the Fedora project So it's nice to be back, it's my first flock So it's nice to be experiencing it This time you've chosen to come to Ireland So thanks Yeah, three years, 11 months, 25 days Too long to be together, but we are and that's the main thing In the Fedora infrastructure team there's a lot of work that gets done This is just a very quick overview of the amount of work that the team actually powers through So there's been over 3,000 tickets closed from last flock to this, like that's averaging about a thousand dollars a year Crazy numbers, but that just proves how dedicated the team is and how determined they are and helpful with Fedora to support the infrastructure that it's built on Out of that, we do have 16 tickets that came through as requests that actually ended up turning into the initiatives that they were much bigger than actually one, two people for a week, etc So they got passed to me In our Centus Infra So for one man band, he's getting through over a thousand tickets in like a year, which is good going I'm sure there's more, but there's a lot of work that gets through and from there we have four tickets to release initiatives as well Release engineering, again these teams are like we're talking single digits for people There's no double digits, the team itself is made up of double digits, I think we're 20 odd strong, but when we're talking about Fedora and for Centus Infra, release engineering, etc it's a much smaller subset of people So to see them in the thousands and the ticket closures is pretty impressive What does it all good mean? So a bunch of times the things get reported and by the time we look into it, we discard it all, it's recorded by itself, magic sometimes exists in the real world We had 146 magic stuff happening, issues that fixed themselves over the course of the last three years, from the last 10% to this one, that's what it means Thank you Finally then we have the initiatives teams, since I joined CPE about three odd years ago, we've managed to get through 14 initiatives, which is pretty good going but we are always open for people who if they'd like to submit an idea for work that we could potentially do, you can submit it in, there's a repo on Pagier, and we always proof the initiatives before we started and you can find that on the Read the Docs, they're called Fedora Arc Team We won't dwell too long on it We'll see you in a bit Thank you Do you want to talk to the projects part? Yep Thanks, Ifa. These are the projects that the community makes use of, and the community of contributors that we're talking about sure, we do provide our offerings, we are websites, but there's a bunch of things that goes into it, right? Things like building the packages or, well, we were just discussing with Justin outside how we plan on undertaking or we plan on maintaining, let's just say, so we maintain a total number of 74 projects out of which 52 of them are used only by Fedora project, 9 of them by CentOS project only and 13 of them by both the communities We would like to divide these projects into certain categories and that's how we are able to understand how important these projects are to look into say, if they are very important and we might as well lose our sleep fixing it, or if they're not important at all, maybe we'll come back on Monday and fix it, right? Say, four classes, critical one these are the application and service that we developed a code for and we maintain our infrastructure, really, really important for Fedora Linux to be built and distributed like the way it is, right? Anything happens to them for a longer period of time? Oh boy, we don't have Fedora Linux distributed the way it is right now then the critical two these are the ones that we don't write the code for but it's maintained in our infrastructure, we deploy it, we really want to make sure that it has as much uptime as we can make it then the standard ones are the ones that we develop the code for, we maintain our infrastructure, but yeah, maybe it's not as important as the other ones and finally ecosystem ones, these are the ones that well, they enhance the quality of life for the contributors, but they are, well, let's just say not as important. So critical first class, 20 projects, 12 of which by Fedora project only, three of them by CentOS and five of them used by both the communities and well, like I said most important and well, these projects make use of Python, JavaScript and C, so if you want to participate in maintaining these projects, by all means come we can use all the hands we can get and well, if you are aware of these technologies then well, I'll say that we will make a great team. Right, so which projects do you think would fall under critical first class?  We have one hand raised. Come again? Well, let's find it out. Oh, both he is actually on let's say. Awesome. Yeah, we could throw it but we don't know if it's going to make it to you. Right, so we have these applications that are used by only Fedora project the list that's only used by CentOS and then the ones that are used by both the communities. Go ahead. Come again? Sorry. Oh, okay, right. Cool. Wow. I mean, let's just say that it probably includes both of them. Right. Yeah. Just a listener and not the results so I think it should include the both of them. Yeah, Kojik might come into critical too, I think. Because if we can't yeah, how can we push stuff out there, right? Where does the updates come from? Unless the updates are coming from somewhere else that we don't know of. Right, so these are the statistics for the technologies that we consider for applications. You know, if they have test cases, how many of them have them messaging schemas? How many of these have documentation and as you can see that the numbers are something that could use some assistance and say if you are really looking forward to contribute, I mean not just in the code base but in documentation as well, you're totally welcome to do so. Right, so a question time. What is the purpose of Fedora messaging? Does it help Fedora project contributors to communicate with each other? Or for them to communicate with Linux users? Or for them to keep track of the important contributions that the community members make and finally, to assist each other to send messages when they are really thankful about it. So let's see some hands if anyone knows about it. Let's find it out. It is indeed see. That's one good thing. Right, so it is see moving on to the critical applications too and 21 in total 10 of them used by Fedora project only, 4 of them only used by CentOS project and 10 of them used by both the communities. Right, so is Koji going to make the list? I wonder. Any guesses which project do you think would fall under this critical second class category? Let's find it out. Any more guesses by the way in the meanwhile? Go ahead. Koji is actually used by both the communities. Look at the bottom of the list. So Koji is there and it is my fault. We kind of have to award them. That's fine. I have to. We could actually throw them. There you go. Perfect. So these are the applications that we consider critical to. These are the ones used in Fedora project and CentOS project as well as both the communities. Right, so now that we are speaking too much about Koji, is it used for making Koji and comfortable homes by the KD contributors? Is it what it is used for? Is it used to make fermented rice pastries for the Fedora project contributors? Or is it used to make RPM packages from the source specification that's fed by the packages? So finally is it the official build system of the Fedora project and all the downstream communities? What answer is the most fitting for the purpose of Koji? Anyone? Come again. D. Right. Say, is it D? Yes it is. Right, great. This was a little twist because both C and D are pretty much the neighboring options, but Koji does a lot more than just RPM packages. It can build OSMHs, bunch more stuff. D it is. Right, now we're coming down to something that's a lot less important, the standard applications. These are the ones that the team write the code base for, but let's just say that they're not as important as the critical first class, critical second class are. 17 projects in total, 14 of which are used by Fedora project, two of which are used by CentOS project and one used by both of them. So, just because the importance is lowered. Okay, let's, we'll allow you to make a guess. What's not so important but very important? Efferman? Okay, let's find it out. Hmm, do we see Efferman in the list? Yeah. No, it's not, right? You'll have to give us something. Yeah, we could have waited for some other time, but then again these are the list of applications that are in the standard category and yeah, you see that these are the ones maintained by the Fedora project, used by the Fedora project and the others as well as both the communities. Well, what's the use of Fedora plan and now that we maintained, well, mentioned of it in the standard first class, is it like to, you know, official fork of Pandora, but Pandora is an official keyword so we can't quite use that name no more. Go ahead. Right, this is a truck running people over and then we get teleported to Fedora plan it. Yeah. Fedora plan it. Okay, just to be clear, we don't endorse getting run over by a truck. Yeah. Which one is it? Let's see some hands. We can't just echo in A, right? We don't have that many swags. Oh, you had the opportunity. We won't let you get another one. Yeah. Anyone else? No. That's wrong. Yeah. Well, let's just say that we have one back for you. Oh, yeah, back in the day. Yeah, no, it's definitely not B. It's A. So, yeah, that's the correct answer. Say one for Samantha. There you go. Say the last part is the ecosystem applications then reach the quality of life for the contributors. It seems like CentOS project contributors are so motivated. They're so motivated they don't need those apps no more, right? So, yeah, let's hear it. Go ahead. Let's find it out. Okay, let's have some more answers before we move on to the next slide. Badges. Badges? All apps are important. Right, there you go. Good. Well, let's hope. So, do we see badges here? Oh, probably sure. Do we see kernel test? Oh, we see that as well. Sorry, what did he say? Open QA. Oh, we don't see it here, do we? Let's just take that as an outcome for this talk, right? Yeah, so we get one for that. Oh, Nick got one, yeah, nothing for him. Right, so those are ecosystem application, right? What's the purpose of meatbot logs? Do you get some artificially oak logs cut it into small pieces and print the, I don't know, the artwork of Swordbot? Is that what it's used for? You know. Yeah, I'm going to let you make that pull request. Right. Or is it for people to catch up on their recordings because, well, we like to have meetings and video calls, is that what it's for? Or finally for people to catch up when meetings happen exactly in midnight, not a minute before, not a minute after. Let's see some hands. Good. Is it B? Let's find it out. It is indeed B. Right, cool, awesome. Let's find it.  But yeah, that was all about it from our side. Thank you so much for being here. And do you have any questions? Oh, Sandro, that's one good. Question depends a bit on who made the slides, but I suppose it was a contribution or consideration between the two of you. What the question is for Akash, what happened to the footnotes? You know, the internet connection is slow here, so we're still waiting for the footnotes to load. Trust me, it's here. You just don't see it because it's not loaded yet. Right, any more questions? Go ahead. I just wanted to kind of put in a plug for Kevin's talk or workshop on Friday. Is it? Fedora infrastructure just CPE. There is people from the community can be involved too. I've been involved with infrastructure for a long time since Mike McGrath did it, which what, that's been a while now. So there's going to be a infrastructure onboarding workshop on Friday afternoon, I believe it is. I just wanted to mention that if anybody is interested in getting involved, come join us. Most definitely, yeah. That session is something that you really want to join if you work for Fedora infrastructure or you are looking forward to working with them. Any more questions, comments? Good. Careful. Where are we with getting rid of some old things? We still have the old message burst. What was the other one? We still have PDC. How are we doing with getting rid of things? Careful, Kevin. We are getting rid of things, we are doing it silently so that no one knows who got rid of it. One of those things that we took care of the last month was nuance here. It's the supplemental wall paper voting thing. We had it around for a while but we just figured that no one used it since 32, I guess. Correct me if I'm mistaken. We decided to ask folks who predominantly use it, design team and other folks if they want to use it. They wanted their things back. We are trying to get rid of stuff slowly. We are getting there. I think one of the things that we are really trying to get rid of is Fed message. Certain applications make use of it like badges so we are really trying to build it to not use it. There is Fed or messaging scheme available if you are an application maintainer and you are still dependent on Fed message. I would encourage you to make use of those schemas, import them into your application, hook it up into the new FMN service that's available that the team worked hard on and is now live. That will allow us to close off the older Fed or messaging notification service too. Plug for Fed message or Fed or messaging schemas. If you are using Fed or messaging or if you plan on using Fed or messaging please write schema. I don't think we have any more questions and I think we are at the top of the hour right now so thank you folks for joining again. Hi, everyone. Hope you had good lunch. Now the next session is on using AIML to process automated test result from open QA from Timflink. As you said, I am Tim. I am a big believer in informal presentations. If you have questions, please ask them. I was a little unclear on I am still going to find out whether I put too much information, not enough information. If you have questions, please feel free to raise your hand. If I don't see you, please interrupt me. That's fine. I don't really care. The whole point of this is to communicate and sometimes that's the best way. What I am looking to do is do a bit of introduction, talk a bit about neural networks, go into the experiments or the experiment that we have done, where we are looking to go in the future and leave some time at the end for some questions. Out of curiosity, is anyone coming in here hoping that GPT was going to be making the presentation? Just me. I was kind of hoping, but yeah, it didn't actually work. Just to kind of get started because it's a very popular term right now and in my opinion there's a decent amount of misunderstanding of what it all can cover. What I am looking to talk about is in general an experiment that we did to figure out whether or not it's going to even be possible to be using AML to do triage. And it is in particular using data from tests that are visual. And I just want to make that distinction because a lot of testing is done primarily in the text domain. And the techniques that are done there don't necessarily work with visually oriented test and vice versa. Just to be clear, it's not about, we haven't created this magical tool to automatically triage test failures. Maybe someday, but we aren't actually to that point yet. And nor will, in my opinion, what I'm about to talk about be able to completely replace human triages. This is, in my opinion again, a good way to make people more efficient to help make their jobs easier and make their lives easier. But at the end of the day it's going to always rely on some form of human expertise in order to actually get the job done. So the system that these tests are running in is OpenQA. OpenQA is a system that's primarily based on computer vision. What it's really good at is it takes a screenshot of a machine that's running, and it has a little picture and you basically tell it, look on the screen anywhere for this picture of, for example, a button. If you find this picture or this little snippet somewhere in the screen, move the mouse over there, click on it. And you create your test cases based off of that. And consequently, the information that comes out of it is visual. Yes, there is some text that comes out of it, but it's primarily in the forms of screenshots and it also produces videos of the entire run. The system itself started with SUSE. As far as I know, it's almost the way they do all of their automated testing. But it is Infador now and it is a critical part of our release process. Every update in Bode goes through OpenQA and at this point, every brahide update goes through OpenQA as well. It runs a lot of tests. This particular idea, this started sometime last year, there was a chronic issue within OpenQA where tests would fail unexpectedly when Firefox, which was being used for part of the test, would just quit. And the test would fail because Firefox quit. And we tried looking into it. It took a while. When this started, we hadn't found the root cause. So we had a whole bunch of tests that were failing for reasons that we weren't testing for, which is not terribly useful. So the people who were doing the triage would go through. They'd recognize that this test had failed for a relatively common reason and would mark it as for rescheduling. And the test would be rerun and tell it either passed or failed from an issue that wasn't this Firefox just decided to stop working. And it wasn't something that was causing the system to stop or anything like that, but it was overhead on the people doing the triage and they usually have better things to do with their time than deal with a common issue that hopefully we can find a tool to work around. So we came up to this question of can we create, can we train a machine learning system to detect that particular crash and then automatically do the rescheduling. But this kind of comes to a question because I was talking about this issue last year. As it turned out as I was starting in this, the bug was magically fixed. I'm not sure anyone particularly found this bug, you know, what the root cause was, but there was an update and they stopped working or, I mean, they stopped crashing. So, you know, there's a very valid question of, well, why do we care? You know, this is no longer a valid issue. Why is there a point in continuing with this? Because, you know, the issue's been fixed. It's not going to be of direct use. And it's an easy answer. It created a rare situation where creating the data set is cheap. By and large, when you work with machine learning, the really expensive part is creating your data set. And something that I've learned through doing a whole bunch of machine learning is if you don't have to create your own data set, don't. It's a lot of work and it's not fun. And, you know, at that point there was a real question of can this even work? Can we look at the screenshots coming out of OpenQA and, you know, get enough information to start doing triage work? So, this was a cheap way to create the data set to set up for the experiment that didn't, you know, involve me, you know, harassing the triagers to help me tag, you know, different runs until they stop responding to my emails. So, I mean at this point, does that make sense? You know, it's set up as, you know, we have this recurring issue and it's visual. I guess I haven't gotten much further than that, but any questions at this point? All right, so we did try a couple of things. I'm generally a believer that, you know, more simpler is more better, but it didn't work. You know, tried to do OCR on the pictures and run that through a classifier and that failed spectacularly so I'm not even... I didn't write slides for it. But, you know, it was worth trying because it's the easiest possible thing.  You have a bunch of interconnected you can call them neurons that work together to remember and process information. It's generally made up of multiple layers, at least modern ones, but the concept itself is not really that new. It was first proposed in the late 19th century and the first computer-based neural network was done in the 50s. And as time went on, you know, it would be popular in research and then people would realize at least in the 50s that they didn't have the compute power to do it so it would die off for a while. Then, you know, people started talking about it more in, you know, the 70s and then it kind of died off for the same reason. The use of neural networks really didn't start taking off until we had, you know, the growth of in particular GPUs. And the ability to do a whole lot more math a lot more quickly that made this theoretical, oh, we can make a computer thing, you know, work like a brain into something that is a lot more practical and, you know, doable. Roughly I'm trying to, sorry, I'm trying to remember that this is being recorded. So that shows up.  I don't have the time to go too deeply into this, but the basic idea is you have, you know, this in particular is one, but you have your input layer and your output layer. Each of these would generally represent one bit. And you present your input. It is connected to all of the different layers in between through various techniques, and then you get an output. Is the very, very high level of how a neural network would work, or does work. In terms of the types, the, what's called the multi-layer perceptron is very much like the diagram I just showed. It is, it was meant to model, you know, how the brain works and how the neurons in the brain work. It is still very heavily used, but it is tends not to be the primary thing. It tends to be a final layer trying to gather stuff together to produce a, at least in the term, in a classifier sense, a single output. Another is recurrent neural networks. It's usually used for more for natural language processing, and where you have things of indeterminate length. And convolutional neural networks. This is, you know, they really became popular for image classification.  You know, that kind of stuff. And that was state of the art in terms of image classification until about 2019. And the stuff passed then, you know, this is a good question of why, and I'm not covering it because I ended up using a CNN. And the reason for it is data. All the stuff that is newer, the stuff that has been state of the art since 2019, requires, like, millions of pieces of data in order to get that performance. And that's you know, getting back to the theme of this was an experiment, and we wanted to do it cheaply. Millions of data points was not an option. So, a classifier.  I think of it as very similar to a capture. So, you know, you get the, we've all seen these things.  So this is, you know, using a human as a classifier. You know, is this, does this image have a sidewalk? You know, yes or no.  So each one of these pictures, you know, can be classified as contained sidewalk, or does not contain a sidewalk. And, you know, using neural networks is one potential implementation for an image classifier.   You know, it's supposed to learn from that, and then be able to replicate what, basically what you showed it. And that's how a neural network also works, is you give it a bunch of data and the correct answers. You know, it learns from, you know, this was correct, that wasn't correct, and eventually you will, well, you hope you end up with something that can repeat what you've taught it. So, you know, we've got a common term, deep neural networks, it just means more layers. More layers, more computation. And that's really what it comes down to. There's no strict definition from what I've seen. Generally, it's three or more layers to be considered deep. But that's about that. My very quick overview of neural networks makes sense. Does anyone have any questions? Go ahead. I forgot about that. Could you go back a slide? You were saying it's multiple layers. Does that mean multiple passes of the same? Is it multiple passes to determine the image, to classify the image? In retrospect, I probably could have found a better diagram. So in this case, this is a network with three layers. You have your input layer, one hidden layer and an output layer. And that's like more layers would add like more if it was fully connected or something. So it's more process, more steps before it finishes. So it goes from input layer to the next layer to the next layer and eventually you get to your output. So it's not more passes, but it's adding more steps before you get from beginning to end. Does that make sense? Thank you. Regarding this diagram, each participant in the first hidden layer would point to all of the points of the consecutive hidden layers. For a fully connected network, yes. So this is a simplistic diagram. In this case, yes, there are cases where you would use that, but when that stopped being state-of-the-art probably in the 90s. So it's a good way to think of it, but it doesn't always happen in this case. Any other questions? All right, so let's get into the experiment. All right, the data set. Like I said before, one of the big advantages of looking at this particular problem is it was easy and cheap to create the data set. We had an issue with the tests and if the triagers saw that test, they would reschedule it. So making a couple quick assumptions that assuming that all of the jobs that failed this reason were rescheduled and assumed that anything that failed and was rescheduled was due to this issue. Are there, did I probably catch a few that weren't supposed to be in there? Do I miss a few? Yes, but over this time period, that holds true enough to produce a valid data set. From that, it's easy to just point some code at the open QA instance, download all the pictures, download all the videos, and we then have our data set that is categorized into two sets. We have jobs that were supposed to be rescheduled and jobs that weren't. And again, I'm harping on the cost because that's one of the biggest things is I didn't have to go to the people doing the triage and after they described the issue and confirmed that I understood, they didn't have to sit there for hours and hours giving me this job number, this job number, this job number, this job number, those were the ones that you were looking for. So the cost in terms of human time was very much minimized. I gathered all the jobs from August 30 to September 13 of last year, grabbed all the screenshots, any text that was produced in the video. Ended up being a little over 31,000 jobs, the full data set is about 80 gigabytes. Eliminating the data brings it down to about half at 102 gigabytes of mostly pictures. So getting, you know, using a convolutional neural network, it doesn't have really a sense of time. You have to give it all of the data at the same time. So I can't just feed it the first screenshot, the second, third, fourth, so on. And what I ended up doing was creating a composite screenshot just as I'm going to go back and forth, but basically this. So this is, you can kind of see the different screens. This is from a part of the job, these are all the screenshots that produced. Here's the first one, the second one, third one, fourth one, so on. So this ends up being one image that contains all of the screenshots and I can feed that into the network all at once. Just getting into some technicalities. Did some sub-sampling in general with neural networks. If you give it something that's rare, I mean this was I think we were seeing maybe 1% fail. I think it was out of the 31,000 jobs, I think it was 900. So I mean it was 1 30th. If you just feed that ratio into a network while you're training it, it's not going to do very well because it can just always say false and you know 29 30ths of the time it's going to be correct. So I just limited the I took all of the samples where it was supposed to be rescheduled and I think I left it at a ratio of about 5 to 1. So for every job that needed to be rescheduled I took 5 random ones that didn't and created the data set out of that. Left the data set as 80% for training so it took 80% of that to do the training did the testing on the rest of the remaining 20% and used a 3 layer deep CNN to do the classification and again this is the composite screenshot that gets fed in. Hopefully this representation will at least not be too confusing. So you were talking about what was it Jeff you asked about you know adding the layers. So in this case this is one layer, this is a second layer, this is a third layer and this ends up being 3 and 4. We can get into technicalities So basically this last one is dimensionality reduction so this is a vector of length 100 and I needed to get a binary answer so it has to be brought down to 1. So that's what this last one is is dimensionality reduction from 100 to 1. Which is why I was saying it depends on how you look at it I'm not going to spend too much time harping on the specifics of what all of this is doing but it is using convolution to steadily go through and convolve the data that starts in the layer and essentially do feature extraction trying to extract information out of it to the next layer. You can it was left purposely without some specific there's a parameter search space that I left for this. The different sizes of the layers the fully connected that last part where it's kind of pulling the information together and eventually reducing the dimensionality what size of the for convolution max pooling has to do with trying to keep it from having all of the information in one part of where it's looking it will take it would take longer than I have to really go through it it's commonly used part it's part of almost every CNN but I did a great search of all of that which is 432 runs and I just want to make a bit of an aside there's been some conversation on some of the fedora lists about do we really need GPUs and I want to point at this where this is the same machine I ran the same network on the same data and it took 4 minutes to run with a GPU and it took 24 minutes to run on just the CPU so it is dramatic and even just with 4 minutes per run that still took more than a day running constantly and it's not, I mean it's a 13th gen i7 with a 24 gig 3090 it's in terms of what people use for machine learning it's tiny but it's I think that says more about the clusters they use for training stuff like GPT which costs millions of dollars but it's still it's the bringing home why we might really need to have some accelerated option for doing the AI, ML and fedora and more of an illustration of like I said the concept of neural network has been around for a while it didn't really take off until we had machines that could do that much extra math very quickly Leaving aside the last one that took hours do you have an estimate of the power consumption of the 4 minutes versus 24 minutes I imagine I could go look at it but I don't off the top of my head be interested to know because oh yeah my my suspicion is that it takes me guessing this is gut is that it's less energy with the GPU but I can figure it out I don't off the top of my head though so one question two slides before if you were to not take the subset, sub-sample if you were to just take composite images how many composite images would you still require to get 80% efficacy throughout for one single failure I don't understand what you mean by 80% efficacy I don't understand for example if you are training for 80% and you sub-sample the dataset if you were to not sub-sample 80% of what I don't understand you sub-sample the split right to 80% to 20% 80% of training no I sub-sampled I can go check the code I don't remember off the top of my head I believe it's 5 to 1 so there are 5 jobs that did not need to be rescheduled for every one that did need to be rescheduled so that and that's just for training purposes if it's too rare like I said you can just the and I've done this where the neural network will just return 0 for everything and you'll end up with like 90 something 80% accuracy because 90% of the jobs were 0 so that's where the sub-sampling comes from in terms of the split it's just 80% so then from took the original sub-sampled it so there was a 5 to 1 ratio of not rescheduled to rescheduled and then from that dataset took 80% of those and that was used for training and the remaining 20% were used for testing and that's it okay and go forward please and so if I understand correctly it took 28 hours to completely train the network? No it took 4 minutes with the GPU it took 4 minutes but I can run it with different parameters so this is just all like to do an exhaustive search of this entire list with 2 tries each was 432 iterations. OK. And like the 28 hours is the number is the total time to exhaustively search through that parameter list to see which one performed the best. Which combination of parameters performed the best. I see. OK. OK. So if you got through this whole experiment, the idea after the 28 hours, you know which neural network configuration is the most efficient for the task. So if you want to train it on a different failure, you wouldn't have to go through the 28 hours again. You could just. Wouldn't even have to go through the four minutes. OK. I mean, once you have the network trained, you can just run it on more. So that four minutes is to train on the 80% and then test on the 20%. So I don't know off the top of my head. I think it would be, I'm not even sure. I don't want to try to do the math in my head. It's much quicker once it's actually trained. Any other questions? So OKQA has configured such that the screenshots are all 1024 by 768. When I did the first try, I took all, you know, did a composite screenshot and then shrank it down to the same size of 1024 by 768. The results were so bad that I did not write them down. And actually, at that point, I was convinced this was not going to work. And I'm like, all right, let's just try this one more time. Let's double the resolution and shrank it down. The composite screenshot, which could be eight times the size of the original, but shrink it down to 2048 by 1536. When I did that, and then did search for the parameter space, it was getting really good numbers. I was going to put a slide in here on precision and recall. So the first thing about accuracy is basically how many, so it got almost 98% of them correct. But the more important things is going through the precision, you know, did it find, I'm trying to think of the, how many of the true positives did it find? And then recall starts looking into how many false positives were there. This is much easier to explain if you include the diagram that I forgot. But suffice it to say, it did well. It is not showing too many false positives, nor is it showing missing things, like things that should have been a positive that should have been a negative. Which in my mind is important for this use case, because if we are trying to rely on something to say, yes, this is triaged in a certain way. This is not triaged in a certain way. If you get a whole bunch of false negatives or false positives, no one's going to trust it. And it's going to cost them more mental energy than if we hadn't done it in the first place. These are that. OK, so there's a typo in my slide. These are the configuration values that yielded the best results. The kernel size is supposed to be 2. I'm not sure why it says 62,000. But that value is supposed to be 2. And getting through it's like, that's great. We have something that's 98% accurate. But is this useful? Is this something that can be applied elsewhere? And this is a way of saying, not really. Like I said, this was an experiment to see if any of this is feasible. But as an analogy, it's like if I have this network and I spent all this time to train it on pictures and I want it to identify cross-box, that's wonderful. But the next capture I get, I need to identify buses. It's not going to find anything. So that's kind of where we're at. This experiment was an analogy to pointing it at open QA and say, find all the sidewalks. Which is great if you're looking for sidewalks, but that's not all the time. And one of the potential issues is that because I had to use the large screenshots, it takes a lot of memory on the GPU. With the 24 gigs of memory I had in that GPU, I could only do the images. I think it was four or five in a batch. And batch sizes are usually larger than that. But there's only so much memory. Do you have a question? And offloading the images from the system is too slow for this application? Folding? Offloading the images from system memory is like if you don't have enough. It has to be on the GPU in order to run. In order to get that time increase from 24 minutes to four, it has to fit within the graphics card's memory. Okay, thank you. Just curious, on the composite image things, as the human behind the curtain here, I happen to know that in classifying these failures, all you actually needed was the failure image. Did you try running the experiment just using, because I believe from the API you can get only the image that the test failed on. I don't know if you looked into that. I did not know that was a thing. Yeah, it knows. It circles it in red in the web UI. And I believe from the API data you can figure out, this is the one the test failed on, and then run it on that. So, I mean, we could, no, I did not know that was possible. So no, I did not run that. I did not know that was a thing you could do with OpenQA. Next time we do it, we can do it that way. It'd be interesting to try my suspicions that would not work. Because, I mean, you're, I've talked about all this OpenQA trioshes. It's mostly you. So, my... You've just been using they, them pronouns for me so far. Well, I... The, my understanding of the time period that this was happening is it's not, you needed more information than just the one that it failed on? No, for this one, Jen, for this one failure, you would know from the screenshot. From the screenshot, you don't, you didn't need any more context of what job it was, what it was doing.  That's the thing the network would have wanted to figure out. But there are other cases where that wouldn't work, though. Just for this specific failure, it probably would have done, I think. We can, it'd be interesting to try. Like I said, my, my incident, I don't want to, I mean, we can get into the details of it. I don't think that would work. Is my instinct, because I don't think there's enough context in that one image to be able to tell, you know, reschedule don't, but it'd be worth trying because it'd be a lot less computationally expensive. Is that another question? This is kind of a question for Adam. Are there things that are successes that look like that failure screen for other kinds of tests? That's a question for Adam, I don't. It's an, he asked, are there things that look like, are there things that would look like the failure screen but would be a success in another test? Again, I think in this specific case, probably no. So, yeah, as I said in the previous question, but just in case anyone didn't catch it, in this specific case, when the test failed, what we would normally see is, instead of something in a web browser, which is what the test was expecting to see, we would suddenly drop to a console and there would be, you know, when this was actually running X for stupid reasons. So you would have the X server output on the console and probably like a cutoff stream of error messages from X usually. So that's not something that's a success in any other case. Yeah, there's no point at which we want that. So I think it would probably have been fairly clear cut, but for instance, there's another problem I have right now where, if we were only looking at the failure screenshot, that wouldn't work and you probably would need this composite screenshot thing, which is an ingenious approach that I like. Yeah, I just, I'm going back to, like I experimented with OCR and the OCR had so many problems with identifying the text correctly, that was part of my suspicion, but it'd be worth trying. Like I said, more simpler and more better. Hey, so can you just go back to the experiment slide? Like the experiment method, where you did like the... This, the experiment design? Well, the RIT search one, where you mentioned that you performed the RIT, yeah, so in the RIT search, so the way you did what I understand, like for every hyperparameter combination, you trained the network using the training set, and then got the accuracy of whatever you were using the test set and continued that for all your combinations and selected the best one based on the test set performance, is it? Yes, that is correct. But do you think that would kind of like overfit the like or influence your network or the results? That's because test set in general is supposed to be kept separate and at the very end, you need to do that. So maybe a split of, let's say, training, validation and test, so that at the very end, when you select the best hyperparameter, then you apply test at the very final. No, you're correct. And to be honest, I didn't do this as formally as I could have. If I was, you know, especially if I was trying to get this published, I would have to do that because you're right. I'm kind of cheating is a way to put it. I think for this particular context for the fact that we're not trying to produce something that's going to go into production, it's just a question of, you know, is there enough information in these screenshots to try and triage? For specifically that purpose, I think it doesn't matter, but you have a, you're correct, I should have done that. But that answer your question? I'm not trying to say, I mean, I think these results are valid, No, I think that's valid. That's because you have quite a lot of data points. So essentially, it's unlikely to overfit, I guess. But, I mean, and now that you mentioned it, I'm curious because I did record all of the, I mean, getting into more specific things, you know, within that 80%, I think I did another 80, 20 split. So it's only using, so I used that 20 for validation. I did record that, those, the results from the validation parts of it. And I, now that you've mentioned it, I'm curious to see if it would have picked the same hyperparameter combination from the validation set versus the test set. Yeah, thanks. Does that make sense to everyone else? What I just said? And what he was asking? Enough anyways. All right. He basically pointed out that there were some flaws. So there were some flaws in my approach that certainly would not pass muster. If I was trying to get it published or something like that. All right, getting back to, and I keep harping on this because I got some questions about, you know, when is this going to go into production? And it, you know, again, getting back to, it's not going to go into production. It wasn't ever meant to go into production. This was an initial experiment. And the idea was to, is there enough information in these screenshots to do triage just based on throwing them at a neural network? And I think that the answer is yes, it's certainly possible and certainly worth looking into. So data and code. I forgot to push my code public because I found a bug last night and something that doesn't affect stuff but I need to push that public. The data is available. It is a 50 gigabyte tar ball that took four hours for my computer to compress down from a hundred and, yeah, 108 to 50. So you're welcome to grab it. It is a lot of data. And I can publish these slides afterwards. If you'd want to see the code sooner then I get around to pushing it. Just come find me. I'm not trying to, you know, hide it. It's just a, trying to make sure it's correct before it gets pushed. Yeah. Hold on a second. Can you wait up for the? So user conclusion is, yeah, it's possible. So if I would love to do a new classifier to find the buses. So, like, yeah, initial around a lot of investigation but the second time when you want to do a classifier for the buses, how long it would take you and how much time it save you during regular QE. The primary flaw in the approach that I've talked about is that it cannot scale.  It'd be nice to be able to detect. In order, the problem is creating the data set because in order for, you know, this kind of an approach to work, you have to have, you know, at least a thousand labeled instances of it happening and then, you know, do all that kind of stuff. And if we're trying for every single issue to find a thousand examples of it before we start recognizing it. So how long it would take you to create new data set for the buses? That's a, it's a hard question to answer. It depends on how quickly, you know, how frequent the error is. The long-haul intent is the person who is going through marking things as, yes, this is related, no, this isn't. So I don't know if I am misunderstanding your question. I have no idea. Like even the scale, like is it minutes, days, weeks, years? Again, it comes down to the issue. In terms of once the data set is created, it's gonna be the same kind of process of, you know, say a day or two of exploring, you know, of computer time to explore the parameter space and then four minutes to train each would be my suspicion. Assuming everything's about the same size, but the most expensive part of this has nothing, it is not the neural network. It's not the time coding it. It is creating that data set. And so in order to get, you know, 1,000 examples of that particular issue, however often it's happening.  And, you know, is, do you need 1,000 examples? It, that's a hard question to answer. It depends on the issue, but what I'm trying to get at is the expensive part is the data set creation and collecting all of that. So let's look at this from the perspective of recognizing failures that, let's say we don't get them in OpenQA. We can look at the Fedora, a more general view on Fedora. So for example, we have stack traces. And we have a system that from the user's systems uploads these stack traces and we have collection of the stack traces. So for example, this can be used if we marked a certain stack trace that this is real error. This, and we know a bug that is associated with. We can have the information similar to that stack trace identified by similar network run to be belonging to the same bug. So kind of how long would it take to achieve this? This is probably something that Miro is asking. Before I answer the question you asked, I just wanna answer a different question that was implied. There is, that's a whole field of research in itself that is not terribly related to exactly what I talked about. But to answer the question you asked, it honestly, it starts getting a little outside of my knowledge. The problem with honestly, every machine learning problem is data. And it's just kind of getting back to the data set. And is it how long? I don't. The reason why I'm taking the stack traces as an example because they are more confined and easier to identify similarity there. Remember when all of these GPT became sort of public available for experiments, but not really run in open source fashion. What people did, they started looking into stable division, diffusion and similar network systems and started to convert other types of information into the visual. So effectively the beginning of that story was that they converted audio into the video, into the pictures and ran stable diffusion to generate something new with those audio and then converted back to the audio. And this is effectively what I am implying here. Now we can take these stack traces, convert into pictures that you analyze, stack them together and run some sort of analyzer. If we know that this particular stack trace, for example, corresponds to the real problem that we already been getting in past and we have collection of those problems actually classified everywhere. We can take that data that are already associated with the solved bugs and train the system with that to figure out some sort of a similar stack trace happening. It's an applied that to the logs of open QA when the crash actually happens. One of the issues I at least noticed is that with open QA specifically it doesn't always give you that in text form. So like the crashes that it was seeing with X it would show up on the screen and it would not show up in any of the text logs. So yeah, getting back to what you had asked, I don't know, I mean it reminds me of talking to someone who did some interesting research of trying to find malware by converting the bits in a binary to a bitmap and then running visual analysis on that. It's a hammering of it because you're making a much harder, you're trying to solve a much harder problem for something that could probably be solved much easier but it still seems to be where the direction of research is going today. And like I said, this is me, this is outside of my exact realm of knowledge. My instinct is that because, I mean it'd be worth trying but my instinct is that because it's more structured, more information retrieval techniques and less machine learning might be more effective because like I said, the downfall of most machine learning problems is that you need a hundred examples, you need a thousand examples of the same thing before you can train it to find it. And in a lot of cases by the time you have that many, the bug is fixed or it's no longer an issue. And either way, finding someone to identify a hundred duplicates would be difficult. So I don't, am I answering your question? That's an interesting way to do it. I don't know how possible, how likely it is to work. I'm just looking at this from the other perspective. So we have roughly upstreams, then downstreams like Fedora, then downstreams like CentoStream, then downstreams like Rel, which have a span of time before a certain thing comes in. So by the time we missed something in QA in Rel, that problem for sure might have already been seen somewhere upstream, upstream meaning in Fedora and in real upstream of that project. So if we could have had some sort of analysis of these problems we see in upstreams and in Fedora, then at that time to generate these kind of data sets and then reapply them downstreams, we might actually get results that we don't see now in Rel QA or we don't see them in the actual support cases and help there because we see them maybe 12 to 18 months later. No, and that could, yeah, having in a way, I'm paraphrasing what you were getting at but having a way to fingerprint some of those crashes to be able to enter from later that, yeah. Just side note that if you want to look for these traces from, I'm gonna compare it to the different issues already reported, you already have it, it's part of ABRT, FAF, FAF already do that, not based on AI but on the editing distance. I was never too much popular, so if you want to dive into that, you can, it already exists. I'm using the traces example here just because people are aware about ABRT but in reality I would take fragments of the actual execution logs with the variation there like SSSD logs or journal with certain things and focus on those because these are the things that we analyze with support every day and some of them span a lot of time in execution and you do match across multiple logs to find the actual problem, it's not in a single place. So extracting these, correlating them and finding them this way seem to be a bit of promising. I think Steph Walter did this couple years ago with the cockpit and they did find some promising results but again training and then reapplying the results fails because you never find the same problem while in maybe reapplying this across multiple distributions from upstream to downstream might give you actually a repeatability of the problem. That's the reality we see. Customers might see the same issue again and again in downstream 12 to 18 months later after we probably solved this problem in the upstream and this gives you a chance to actually catch something that you forget to back port for example. That's my point, maybe this is a real value here. Okay, and I'm happy to talk about this more but I have five minutes left and we'll see if I get through everything. I should be able to. So getting into, it was an experiment. It was never meant to go into production so what can we do with this? Can we make something out of it? From what I understand and Adam, this is from a conversation we had, something that is possible that I want to take for the next step is can we take those open QA failures and start grouping them by root cause? And so that you don't have to have so much, again trying to revisit this whole thing of making people who are doing the triage work more effective, it takes less of their time because they have other things they need to be doing rather than just triaging the same thing over and over again. I think the value is not so much about taking up people's time. So the Adam W. Restot bot runs maybe every two hours during the day and then there are these eight hour intervals where I have to go sleep and that means that if your update is blocked on one of these failures, you might be waiting eight hours before it gets restarted. If we can do something like this, then your update will get restarted immediately and then we cut out that latency. I'm talking about more general. Not just doesn't need to be restarted as in can we change open QAs so that there's some suggestions up? These five failures look very similar. And trying to group things by root cause and failure not just doesn't need to be reset. Yeah, that could be helpful for sure. So yeah, I agree. I agree it's definitely worth looking down. And I think that the thing that we have to look at is the whole question of how long does it take to get enough data together to classify things versus how long do problems exist? But there are, I mean the X thing lasted for several weeks. This 404 thing has been going on for a couple of weeks now. Yeah, and like I'm talking about something completely different. This is related and not. So the, but the idea is, yeah. It's still the same thing, it's not always the same ones. But one of the things that, I'm sorry to tell you I'm just trying to make sure I finished my slot, finished before time's up. The, there's a system out there called Tango. It's been published in research. Basically it was developed to try and find duplicate results from screen captures of mobile apps. And it used video and text to try and find duplicates within its data set. It's something I do wanna look at. Cause I think it's promising. It's related enough to be worth looking at. But as I'm repeating myself for the 50th time, the problem is data. Modern machine learning stuff needs a lot of data. We have only so many expert hours and only so many emails to set experts before they start ignoring my emails and never talking to me again. So it's a question of this is the real cost. Is this going to look promising enough to spend the time to create the data set to do more research? But one of the things that I also wanna look at is a technique called active learning. Well, I call it technique. An approach called the active learning. And the idea is instead of sending every single sample to an expert to annotate, analyze what your data first and organize things such that you are using them minimally so that you can gain the most amount of information from the least amount of expert time. And that's, so the idea is from here to look at Tango to look at active learning so that we can minimize the amount of time that we need from experts and hopefully find something that can start grouping those failures by root cause with enough confidence that it doesn't end up being noise. So just repeating the conclusion, it's promising but in my opinion but it is only beginning and going forward it's going to start getting expensive. One of the things I do wanna harp on is and I'm leading into this, all of the work that I did was on an Ubuntu machine because this stuff does not run well if at all on Fedora. And we can get into a whole bunch of stuff about why and what can be done. There is kind of a solution coming up that someone's working on, basically the replacement for NVIDIA Docker. You used to be able to pass the GPU through to the container and have all of NVIDIA's proprietary crap in that container and run it on Fedora that way. We stopped shipping Docker. The replacement that can use Podban. I think someone's working on packaging it. But that's a lead up to if you also don't like the fact that this all had to be done on Ubuntu. We're working on stuff to fix that and maybe you should show up for the, there's a, at 4.30 in the room next door, we're gonna do some of a meetup to start talking about some of the stuff and what we're doing to fix it. I think I'm out of time, but questions, comments. I guess if you have, I am out of time. If you have questions or comments, please come find me. Thank you so much, Tim. And thank you for all of you who are here in the Fedora Leads, the Linux distribution development room. We will be picking up again in five more minutes and if you are on your way out for a moment, there is a badge sign on the back desk there. Please stop by and get your flock badge. If you haven't already, you can scan the QR code. So I just wanna remind everybody for that. So we'll be picking up here in about five more minutes. Yep. All right. For those of you who are joining us online and around the world and those of us here at flock today, welcome back. I'm glad that you're here to continue with the Fedora leading Linux distribution development track. It's my pleasure to introduce Michael Cornetchi. Cornetchi. Cornetchi, ah, almost there. And I will hand it over to you for a talk today about what's new in the land of releasemonitoring.org, 2023 edition. Thank you. I was already introduced. So this talk is about what's new in the land of releasemonitoring.org. This is the 2023 edition. As you can see, I adjust the fabulous slides. My name is Michael Cornetchi and I'm mage from releasemonitoring.org. That is not my official role, don't think about it. So first thing first, where is my magic hat? Oh, here it is. Okay. But how we can get it from the slides? Let's see. Oh, it's gone. Where it is? Oh, it's here. Okay, found it. So now I'm the mage. And let's start with a story. So once day I was going to my magic tower. As you can see, it was nice day. Everything sunny. It was like in some kind of paradise. Plenty of flowers. And then I got inside. And I found stairs. So I get up and then going up, I was thinking if what I can see. As you can see, there is a balcony in there. And what do you think I could see from it? Okay, so if nobody knows, it was actually the role of releasemonitoring.org. It's all real. And what it is actually? So there are two parts of the releasemonitoring.org. It's the Anitya and the new hotness. The Anitya, I will show you first because it's just easy to show. Easier to show. Let me just find it. Oh, let me fight with it for a moment. Oh, let me open it. It will be easier. Okay, so here is how it looks. This is the Anitya itself. The Anitya is a front-end for watching for the releases of your favorite projects as you can already read. And this is how it looks. And let's go back to the slides. And what it allows users to add projects to watch for new releases. This will allow you to actually see what projects, what versions are available, what version is the newest, and it will allow you plenty of customization for the projects. It automatically checks for new releases of the upstream projects. You can see what is happening there. You can see hope when the version actually came out. And it sends federal messages to the message bus when the new version is retrieved. So all of this is actually done in the front-end. This is what you can see mostly in the plant of Riesmointing.org. But there is the other, the new hotness. The new hotness is kind of somewhere inside of the Riesmointing.org, but let's say it's floating island above. And what it does, it's listening to messages emitted by Anitya, creates or updates Baxilla when new Ries is built, font, and it can start scratch builds configured. This is what you will get if you are a package and you get notification on Baxilla. It is the issue created by the new hotness. And now we have some magic numbers, who doesn't like numbers for Anitya first. So it will be in this format. I will show you how it was the number in the previous nest and year later and what this is for between the last nest and the new flock. So we will do it a little more interactive. So we had two releases of Anitya two, one year ago, or to the next, to the previous nest. So how many releases do you think we had? It was more, less, more, okay. Yeah, you will be right. So next, we have 266 commits. So do you think we had less commits or more? Yeah, it's less. And it's less because we actually change how the dependencies are solved. So we don't get that much dependency updates than we get before. So plenty of tools were created by both before. And we had 13 contributors before. How many do you think this year? More, no, it's less. But I would say that this is because in the last statistic I probably included the both accounts and those are removed. So the number would be same. And how many issues? Less, less issues? No, it's more. But this means more people are actually using the release monitoring.org, which is great. Okay, and closed issues. Yeah, it's much more, much more than last year. And almost the same number as the created issues, which is nice. And the last version that I presented here was 1.4.1. So how many versions do you think was before? We know that it was six, but what is the version number? It is definitely bigger. So, to that, oh no, 1.1. We had one hot fix and few mineral releases. There wasn't any major change, at least not in the backend. There was some changes in the front end that were actually really big. And how many projects we watch? We had 216,314 last year. So how many? We have no. More. Yeah, definitely more. So 309,000. And how many of those are Federa packages? If 20,600 last year, it's close. We didn't get that much new. But I'm not sure right now how many Federa packages are actually out there, how many Federa packages are actually packaged and maintained right now. So not sure if the number could go that much up. Okay. And now look at what the goblins delivered, so what new features we have. So you will now see the goblins that are delivering things. Those are actually DICE goblins, but this doesn't matter. Let's say that those are the goblins that are working on the RISC 1.0.org and trying to deliver things. I'm one of them. So we had one new thing is that we have sort of the list of distribution in the project view. I can actually show that when I switch to the correct tab. Oh no, okay, here it is. Okay, so if you saw the project previously and see any mappings, it was just how people created them. So we have no way that you can see them alphabetically. So it's much easier to find what you are looking for. And okay, so let's go to the next one. Let me just switch the slideshow. And the next big thing is migrating to bootstrap five. The previous where previously it was with bootstrap three. And it took a great amount to actually migrate it, but the new front end looks much more clear, much more clean, much more modern. So it took work actually was good and satisfying to see that the outcome is actually something we can look at and we can show the people. The next thing that was related to it and why the migrating actually started was we actually had dependency watching for JavaScript packages. And this was actually one of the things that was found in this, that we have really all bootstrap version. Great, source backend. If you know how the Anintia works, or if you don't, there is plenty of backends you can choose from, I will open the guide. As you can see, there are plenty of them. Those are actually all the Gitforges or sources you can watch for. We added the source hat, that was one of the newest one. And one of them is made custom so you can create your own regex that is actually trying to parse the page and trying to find the version based on regex. So let's go back. We have configuration for this trainings. I will show that because that is really nice feature. If you go to any project, let me just go back. And you can see that there are links in the package names and if you click on them, you will get redirected to the repository in the distribution. This was originally created just for few distributions and those few distribution were just having it hard coded in the web page. So the change is that it is no configurable. There is in Anita configuration you can actually set distribution and the link. And it just adds the name of the package that is specified in the package name. So you can use it for anything. You can see here is a Fedora. So let's just look at the Fedora one. And it will show you where it is, how it looks. And it's working for the FlatHub as well. So yeah, it's usable if you just want to know what distribution are actually using it. You must take in account that all of these are actually edited by somebody. There is no automation that is creating the package mappings. So if anybody from the distribution wants to update, it's on you to actually add the correct project. Okay, so let's continue. And we have another bagend. This is the CGIT bagend, just I think as the last one. And the Googs bagend. And now we will look into the crystal ball and we will see what the future holds. So one of the things that we, that is actually wanted by plenty of people, I closed plenty of duplicates for this one, is to have some kind of replacement rules for versions. If you get a version right now, only thing you can do is to remove prefix from the version. And there are some, let's say projects that actually creating really strange version strings. And this is to help them to actually just find the version in it. So kind of reg X that will just, you will set, here is the major version, we use the minor, we use the what fix version or something like that. And the rest will be just thrown out. So the versions are much clearer. Support for different version streams. This is something that is related to both Anidya and Newhotness. The different version streams means that we will have, we will watch separately the versions for 0, 1.0 release via 2.0 release and so on. There's plenty of people actually wanted to get notifications about the older versions, even if there is a newer. So this will help with that. A new authentication backend. This is ongoing work. This is already being done partially. We will go from the Flask OIDC to Outlib. And we already have some work done for Google and GitHub, but we need to add the Outlib support for Fedora. Because the Flask OIDC is using deprecated authentication number, which is not great. But yeah, this is something I need to look into and didn't get to it. And the next is RSS feeds for project. So if you just want to watch one project, we would like to have option for you just to add it as RSS feed and it will notify you when there is a new version. And automatic filling of project details when creating a new project. This is something, if you are familiar with ProtonDB, for example, if you just put there the ID of the game and it will show you all the things that needs to be filled in. So it will just prefill it. And I would like to have something like this for Anitya. You will just put it the URL of the project and it will prefill most of the things for the project. And now some magic numbers for the new hotness. So make it interactive again. So we had four RSS till last nest for the new hotness. Who many do think there was this year? Less or more? More. More? No, it is actually less. And who many commits? Less, more? More. More? No, it's much less. The reason for that is, like I said, in case of Anitya, we changed how the dependencies are resolved. That is one of the things. And the second thing that is actually bank here is that we actually don't have that much issues on the new hotness. Which is another thing. How many contributors do you think we have for the new hotness? We had five. Less? Yeah, we have three. But as I said, this could be the both accounts. I forgot to remove on the last year. Issues created. We had 62 last year. 24. And how many closed issues? More or less? It was less. But because we didn't have that much open issues. So. And version? Do you think the version is bigger or less? No, it's not. I don't think it's the same. Could you want to actually guess what version do we have right now? It's fine 2.24. 1.2.4. And the fourth one is, this is actually because there was only a few hot fixes that needed to be done. And otherwise I didn't need to touch it. There are a few feature requests, but they are not high priority feature request. And for the issues, it's actually, when the issue is reported, it is closed really soon, because it is much better to maintain it and much more stable, it's easier to find what is causing it and fixing it. There are a few that we didn't actually found out what is causing them, but this is something like a heisenbach, so it's hard to find. And what features? Okay, I will ask. How many features do you think there was delivered in the new hotness? Okay, yeah, it's zero. We didn't have any new features, but as I said, it looks like the new hotness is right now stable. There are a few features I'm still waiting for Pagur update to happen for the diskit, for the diskit to actually be useful, because I already have in new hotness option to watch for all new releases, not looking only at the new best one, and option to watch only for the stable releases. But both of those options are dependent on diskit to have new version of Pagur, which didn't happen yet. And what is the future? So I would like to have support for flood packs, for Fedora flood packs. So we will just create new bugs in the Fedora flood pack namespace. Same for Apple. And that is actually for all. So any questions? Any questions about head? No. No, it's not about head. I'll just give him the microphone, so. I know, I just wanted to say your head is fascinating. I mean, do we have any more pins in a flock? This year? Because I would like some pins on the head. I think that they do have some pins for giveaways later. So if you're in some of the social nights, I think they have some Fedora pins. Okay, that sounds great. Okay, so the question was where did I get it? So I ordered it from some eShop. I don't remember. And I liked that the colors are actually in Fedora, white and blue. So it's nice. And any other question? So forgive me if I'm just oblivious. I don't think there's anything in it for like CVE tracking separate from just actual updates. Have you considered being able to like look for CVE fixes and stuff as well? Oh yeah, that is one of the things the CVE fixes are not part of the Anitya. Anitya is just watching for new releases. It doesn't know what they seem to release. In case of GitHub, you can look at the commit that is related to release. So this could be used. You can look at the upstream, but you need to look at the locks of the project to actually see the patches. Yeah, I was more or less just asking is that a feature that's been considered for the future is just automating some of that? At least flagging saying, hey, there's a CV on this. You might wanna go look for fixes. Yeah. Okay, in case of Anitya, Anitya doesn't look for this. I'm not sure if we plan it as which I don't know if we even have issue for it. Let's just kind of go basically. Yeah. Okay. Just right now it's just looking for new releases and the new hotness is the one that is creating the bugs. So it would need this information from somewhere and right now we don't have it, especially for some of the baggants that don't actually share it. We have custom baggants, it's usually just parsing some sites so we don't know what is on the site. Just looking for the version that is new. Okay, so here are some links. I'm not sure where I can share the slides actually. You can put it on the sketch page. Ah, okay. Okay, so I will put it there. And yeah, you can look at those. There is blog post. Oh, okay. Well, it's really a personal interest question because I put in an issue about supporting certain source force projects that release like different streams of artifacts that are tracked independently, they versioned independently. So they like sub-projects from one project. Because I got bitten by lack of this feature and because the new hotness filed a bug with the incorrect version that was the defaults, let's say sub-projects version and not the one I was actually pointing to in this particular project. Yeah, in this case it depends if the projects actually have the DAX different for the projects that are tracked on the same repo. Because if the DAX are just version, we can actually. There are sub-folders. Sub-folders, yeah, okay. Yeah, I'm not sure if we can, we don't support that and it's not that much project that's actually using this. So it's a low priority feature in this case. But yeah, it's one of the, I know that there is issue for something like this. Okay, thank you. So if there isn't anything, I will share the slides later. The blog post is something I didn't update it in a long time but it's why I actually have the head and why I'm here as a mage and you can actually read why it is like that. And there are some links to the repositories and for the messaging schemas that are actually very emitted by Anitya and the new hotness. And that is all. So thank you. Thank you very much, Michael. And if you all are welcome to head downstairs for coffee and tea, there's a social breakdown stairs and we'll be picking it back up in 30 more minutes. And if you haven't already gotten your flock badge, there is a sign on the back of the table there where you can scan the QR code and get a fedora badge for flock. So we'll see you all back in 30 minutes. Thank you so much. All right. All right, well welcome back everyone to the fedora leads and Linux distribution development track here at flock. It's my pleasure today to introduce Adam Williamson who will be doing a presentation today, Fedora CI and automated testing. I'll hand it over to you. Okay, thanks a lot. So yeah, I'm Adam Williamson. I have the Fedora QA team lead for Red Hat. I've been doing this for quite a long time. I started in 2009. As you can see from the next slide in my deck, which is a picture Mo Duffy took in 2010 in Zurich. So there I am, you know, young and optimistic with Peter Robinson and I think Jesse Keating. So yeah, I've been doing this for a while. Just so you guys know, my policy for talks is to hold Q&A and feedback to the end. Otherwise it's hard to get through on time. So I'll definitely make sure to leave space for questions, but just hold them to the end. So yeah, let's get into it. So next slide, I guess that's the next slide. So how do we test Fedora? This is kind of some background. This is all gonna come up later on. So I just wanna establish it. There's different levels you can test that. There's disk it, which is where we keep the sources for the packages. So you can actually run tests before an official build has ever happened. You can run tests on pull requests. You can run nightly tests on the Git repo if you like. You can test after an official build happens in Koji or an unofficial build, you know, any kind of building Koji. That's another place you can test. You can test updates, which are when we put related builds together into an update and say this is the thing that needs to go into the distribution, you can test at that level. And you can test composes. And composes are when we put all the builds together and we make images out of them, we make repositories out of them, we do that nightly for most streams of Fedora. You can run tests after composes happen. So these are kind of the main levels that you can run testing at. So some background to, I mean, the topic of the talk is where we are with automated testing, but you know, how we got there in the first place is kind of useful for the story. So in the oldie days, this is what we did. We basically, in a QA, consents context, we didn't do any testing of package sources. Like maintainers might do it for themselves, but that was outside of the scope of QA. Build testing at the build level, again, we really only had what maintainers did themselves in the check section of the package where a package build can run test scripts and fail if they don't work. At the update level, we had manual testing.  There was manual testing of updates, which was kind of haphazard. We have this little mechanism where if a wiki tests in the wiki, especially tagged, it will show up on the update page. So we can kind of give you some guidance. This is a test it might be useful to run on this update, but that was about it. Other than that, people would test the update and say, yeah, it's good. What did they actually test about it? We don't know. They just said, yeah, it's good. Or they didn't, if they said it was bad, they would normally tell us why, which is useful. But yeah, it's very haphazard. There's no way of knowing that what you're testing from update to update is the same. And sometimes an update just wouldn't get any testing. So do we know if it works? No, we don't. We have no idea. And it's very difficult manually to check if an update breaks initial install of the system or the compose process. And we'll get to that later. This is an advantage of, but that was a thing that was very difficult to do with manual testing. For compose testing, that was probably where we used to focus our manual testing the most. So we would put a lot of work into manually doing what we called release validation testing, which was, it's focused on, okay, we have a release candidate. Is this release candidate ready to go? And we would run over a hundred manual tests to decide if that was the case. And that gave us pretty decent coverage. We didn't ship many completely broken releases, but it's very time intensive.  Camille remembers these days when we would just spend weeks just spin up a virtual machine, run and install this way. Does it work? Okay, do it again. Do it very slightly differently. Do this until you're tired of life. Yeah, so that was an issue. And even the amount of resources we put into it, we couldn't test every compose. There's a compose every day. We would go insane trying to do that. So that was another limitation. The consequence of only being able to test periodically is that you get problems where there's a bug and then by the time you've noticed the bug, there's another bug. So you have these kinds of stacks of bugs. By the time you get to testing a compose, there's these four or five different bugs in it, all of which you could have caught one at a time if you could test every compose. And it's, if you don't test, one of the nice things about testing very frequently is that it's easier to identify what broke something. So if you don't test very frequently, you're looking, say we ran a test on a compose and then we do another test two weeks later and we see a problem. We know that something that changed in the last two weeks caused the problem, which is better than nothing, but that's a lot of things to check. So there's a lot of issues with only doing manual testing. Just some quick sort of history of how we used to do, how we did validation testing and where we got to before we started automating. On the left here, it's, I have to shrink these down a bit, but this is the earliest organized QA testing of Fedora that I could identify as from Fedora core five in God knows when that was 2001 or something. I don't know. There were 26 tests, which I counted out of this table. So then by the time we got to, oh, 2006 I have the note here. By the time we got to Fedora 21 in 2014, which was the release before we started automating things, basically we had 138 tests. That's not all of them. That's just some of them that had to be done manually for every compose, which was that was a lot of work. So we'd gone up six times and that still wasn't testing everything and it was taking up a lot of our resources. So that was the point of which we were like, okay, we need to change something or we're all gonna go insane. So that's kind of where we got up to. And just as a side note, after we started automating things, we've continued to grow the test set. So as of 2023, the Fedora 39 matrices have 202 tests on them as I counted. This is, I've done this talk three times before for anyone who doesn't know. I have a bit longer today and I'm sorry, I can be a bit goofier. And this morning I found a talk that Will Woods did at FudCon Toronto in 2009. And if I'd had a little more time, I was gonna rewrite this entire talk to do it off Will's slide deck, which would be great, but they didn't have the time. So instead I've kind of dumped a few of his slides into my talk just for fun. I don't know the license on these slides, unfortunately. I asked Will, but he hasn't got back to me yet. I'll update that later. So this is from 2009 from Will's slide deck when he was, this is when they were starting AutoQA, which was the automated testing thing at the time. And the fun part is a lot of this stuff is relevant. This is the stuff that 14 years later we have finally managed to fix. So they were discussing, okay, what makes raw hide broken or good? Can we write code to check that? How can we run that code automatically? All of these are the things we have finally managed to get around to solving an automation recently. So this is all the stuff that we've actually managed to implement. It took a while, but we got there. So what do we have now? Where did we get to? What are the automated test systems in Fedora? What are they testing? That's, this is the main meat of where we're getting to. The two main automated test systems are Fedora CI and OpenQA. The point I wanna get across here is why do we have two? What are they both doing? How do they complement each other? So this has evolved over some time. OpenQA is, was effectively came out of that problem of having to do a lot of manual testing, but still not being anywhere near the amount of testing we wanted to get done. At that point in history, AutoQA had evolved into something called Taskatron, and we were working on that, but it wasn't at the point of being able to automate all of those tests for us, and we wanted to do something. So we picked up this tool called OpenQA that the SUS folks wrote and said, look, this can, we can use this right now to help us, we're just gonna pick this up and use it. That was basically a Skunkworks project which ran on someone's computer under a desk. It ran on SUS at first. So that's where OpenQA came out of, and we've just kind of gradually built it up since there. Fedora CI is a more kind of planned effort but had more resources behind it. It had some involvement from the REL side of things. It was kind of an idea to standardize and share how automated testing could work across Fedora, CentOS, REL, and make it so that RedHack can push some of its testing that it was doing internally upstream. So try and provide a kind of shared environment so we can get some of those tests running outside of the RedHack firewall. And Fedora CI is kind of trying to offer tools and workflows to packages in a form that will be kind of familiar to people who are used to working with CI systems upstream, whereas OpenQA is very different from that. So that's kind of the background where we have here. So the way it's worked out is that CI is kind of intended to be self-service. If you're a Fedora packageer, it's like it's a way you can write tests for your thing and get them run at a time in a way that makes sense to you. You can have the tests run on pull requests. You can have the tests run on package builds. The results can come back to you in Diskit or in Bodi. It's not, there's nobody centrally writing the Fedora CI tests in general. That's not how Fedora CI works. So it's kind of, it's a developer workflow is the idea. As I say there, think about providing CI services to people working in Fedora. OpenQA is more, it's testing of Fedora itself. It's not something for Fedora people to use to test their little bit of Fedora that they're working on. It's about making sure Fedora, the thing we define as Fedora is working well. With Fedora CI, the system is the thing. The people who work on CI wake up every day and think about how they can make the system better. With OpenQA, the system isn't the interesting thing. I'm the, me and Lucas are the main people working on OpenQA. We kind of wake up and say are any of the tests failing? How can we fix that? Can we write some interesting new tests? We don't think about improving the test system. So that's kind of the difference there. So get into some more detail on both the test systems. What is OpenQA? OpenQA is a system for testing which it's kind of designed to test the way a human tests. So OpenQA spins up a virtual machine and then it knows how to type. It knows how to see things on the screen. It knows how to click on them. And all of this is happening basically at the level of the virtual machine. It has no idea what operating system it's testing. You can use it to test a firmware interface. You can test Windows. It doesn't hook into any kind of toolkits or anything like that. It's just looking at the screen clicking on things and typing things. And then, yeah, the way it originally works it's all about looking at the screen, matching screenshots. It's been developed these days so that it does have some integration for Linux consoles specifically. So you can run commands at a console. You can get the output from them. You can get the exit code from them in a kind of programmatic way. So you can write tests that are like, okay, run this command, did it pass or fail? That can be a requirement. So it has that capability. What we use it for, as I said on the previous slide, we use it for high level functional testing. It's like we define, we kind of build off the release criteria and things like that. And we say, okay, what are the core requirements for Fedora? What are the things we want to know at all times are these still working? That's what OpenQA focuses on testing. So we run a set of tests on all Fedora composes, also Fedora CoreOS composes, IoT composes, cloud composes, there's probably some other kind of compose, I'm forgetting. Any kind of Fedora compose, we run some tests on it. ELN we're testing now, w