se are generic stuff and they run even via containers so because not every test needs a VM right if you don't need we can run against containers Zool is the next CI system for testing we have gith merge gith PRs on Pager and we have it actually in two flavors it's also for CentOS stream this gith contributions there we integrate with Zool via playbook once you have the HTTP API you can integrate wherever you want and we are providing the results Zool is providing the results directly from merge request maybe you know Zool check it out yeah very easy onboarding it's configurable this is not configurable so much as it should be and I will go to the demo then we have a gith collections workflow you can read what are the benefits but rather you spec it if you don't need secrets with the github actions you can use secrets and if you really need to integrate because are you right to hack or call API directly you can use our HTTP API directly via HTTP or whatever or you can use testing form CLI tool and in future we will have multi-host tests and easy onboarding via Fedora and Retta single sign-on and more infrastructure grow more infrastructure because we are hungry and now onboard if you want there is onboarding guide and now demo time so I want to this is like the newest thing I wanted to show you it's available if you are willing to share public IP with testing form then I can make it available for Retta but it will take us a little bit more time to share the public IP automatically so you don't need to know but if you want only onboarding let me know somewhere you can find the contacts in the last slide so I want to reserve what? I want to reserve sent to stream okay I want just Fedora Rohite I will just click it Fedora Rohite Intel architecture reserved for 30 minutes and now I open for your questions until these 3 minutes 3.5 minutes and I will get and I can do whatever you want so that's my demo reservations on testing form if you have a token you can reserve a machine and do investigation this gets you the exact same environment as testing form does you might need to pass in some special parameters to make it the same as your request was but maybe we will adjust that it reads it out automatically so this is very cool I'm really happy about it I'm ready for questions it's understandable yes so the question is if I know how many people use this for a packet I think packet has a nice light with those large components right I can query the SQL database but it will take a little bit time so after the well I'm we are testing hundreds of Fedora packages 900 was last I checked but with packet I will need to check is somebody from packet in here maybe they know they didn't come so we will need to find out we don't have good statistics for this and we should have but I can query the database after the no problem sorry any other question yes so if you have some specific hardware requirements you would like to because we own the infrastructure I own AWS account I know beaker so you have a beaker account or AWS account I think it should be possible we currently do only AWS open stack but it's internally publicly do only AWS so if we have accounts in testing farm it's possible then to provision these other clouds we have some Azure support but others are not implemented so if you need for example GCP it needs to be implemented so you can contribute our code is open or you can file us a Red Hat issue and we will try to get to it it's all we are all based on user request so if there are more users asking for GCP then we are planning also to make it possible that you can give your credentials somehow to us then we can use your credentials because that is needed for cloud costs or what not but otherwise we are all running this on our accounts currently internally we have cloud cost dashboard I can tell you that your team how many dollars it spent on AWS if that works for you but I'm free to speak after the talk and we can sort out the details any other questions was it digestible or not that's an improvement last time I said that slow down the recording if you can understand so it was better I promise it will go through just let's wait a few seconds how much? I'm over time 2 seconds 3 seconds now it's actually now it's preparing the environment I be hacked a little bit that it shows the status you will see that it was running setting up and now it's preparing the environment at the end it should log in it takes so much because our guest setup playbooks are slow and they are updating the whole system and rebooting it so that's why it takes longer so maybe we it works and I wanted a normal machine so I have just something but I could provision a machine with 32 gigabytes of RAM more disks more disk size but this works if you want this it's available we have actually this tool on PyPy if you have a token you can already use it just let me know somewhere that we can meet together and I will meet your public IP but next time I will steal it and we will make the access to you because you need to somehow access these AWS resource we cannot just open it for everybody because somebody in test they set the password to Fedora then the bot comes in and it will do something right so we need to make sure that this is safe because it's an internal network so we already VPN so sorry but if you want this it's available catch me thank you for your time yeah perfect welcome to the session about the new version of the NET5 and now the floor is yours Jan, Pavla and the others ok thank you very much I would like to thanks organizers to get the opportunity to give you our excitement about the next generation of the software management in Fedora and other downstream distribution before I start I would like to introduce my colleagues Pavla Kotokhiwola and Honza Kolarik and my name is Jeroz Lamnacek and everyone and we are from DNF team well during the presentation there will be three funny parts with the passing like therefore enjoy it have fun so first so that we are all on the same page and know what we are talking about so DNF is a package manager which means basically software for installing, removing upgrading other software and to put it into context on the lowest level you have RPM which does the actual installation and removal above it there is DNF that uses RPM and does other stuff like it understands repositories and modules and uses Lipsol to resolve dependencies and so on and above it there is packagekit which is DNFDemon and that is used by some graphical user interface tool ok so this presentation is about DNF5 but first let me just shortly talk about the previous version about DNF so that we know why the DNF5 was needed what was the motivation to move to the next version so in here you have the structure of the original DNF so this is not DNF5 yet ok and you can see it's a little bit complicated in the middle there is this big box called flipDNF and it's containing other smaller boxes and it's DNF library and you might think ok it's somehow logically composed of small libraries but it was actually the other way around so first there were these small libraries and it was then meshed together to form the DNF and so you might see how that might be a problem because there remained some duplicities it didn't fit together very nicely there was a lot to work done on it to work properly but still it's not perfect then you can see in the purple boxes there is packageKit, I already talked about it it's the DNFD one then there's DNF, it's the command-line tool then there's microDNF and that's another command-line tool so you might ask why do we need two command-line tools for this and the answer is they're both a little bit different DNF can do a lot of things it has a lot of features but it also needs Python so it's not very suitable to use in containers when you want minimal installation and that's where microDNF comes in but microDNF on the other hand doesn't do everything that DNF can do also that's because they both use different parts of the library so for example when a change is done in DNF it doesn't get automatically done in microDNF again you can see how that can be a problem and last thing you can see there is two plugins and again there's duplicity there there's C plugins and Python plugins and there's overlap in functionality so you can see there's a room for improvement but we got to a point where it was very difficult to make improvements without breaking compatibility so this is where DNF 5 comes in and yay now this is DNF 5 structure and you can already see it's much more simple this big box is just the library it doesn't compose of different sub libraries that don't fit together very well it has completely restructuralized API it's written in C++ and yeah you can see that there's again a DNF 5 beamon that aims to replace package kit and there's only one command line tool which is DNF 5 DNF 5 the command line tool is now great because it has all the features that DNF had but it's just a slight wave of microDNF because it no longer depends on Python but don't worry about Python, we do have Swig bindings so it can still be used via them and it's really great because all of these three things use the same library and not its different parts so one change can be used for all of these as well I think you can still notice that there are two boxes for plugins but that's okay because now these are actually different there are C++ plugins for the library and then there are plugins for the command line interface so for example adding a new command would be there while something that would alter the run of the whole DNF would be in the plugins for the library and it could also be used by the daemon yes? current DNF plugins live in Python with DNF5? current DNF plugins if they will be compatible with DNF5 I believe not yes so they will need to be rewritten but we are rewriting many of plugins right okay I will hand over okay there is another question well the question is whether CLI plugins use SWIC or whether they must be in C++ yet they must be in C++ the reason is quite easy if anyone will write an important plugin for your distribution in the Python then you will depend on the Python and then you will lose one of the advantage of the whole spec therefore for us it is much more better to write or to help you to convert your Python plugin into C++ then to support Python interface for the command line plugins I'm sorry let's wait for the question and answer section with other questions maybe it will be better to have it in the one place sorry well when we talk about improved API that we suppose our new library will provide what it means we try to provide a better workhorse safer workhorse it means that if you run things incorrectly with your new code DNF5 should alert you stop you and explicitly say you do things wrong you should call it in a different order why this is important we have experienced many reports from the users like you know it doesn't work and the simply answer is you run things not according to the description in our one page and so on therefore the workflow was not guarding or even worse no one reports anything and you think everything is working according to your settings for example you modify settings at the time when it makes no sense but then every time when you ask what was the setting for the operation you receive the new value that was not used for the operation therefore that's why we have some locking mechanisms for example for the configurations of course we also improved transaction reports well formally DNF4 use or overly use logging therefore if you have any problems then look to the log but your application is unable to handle such a report because parsing anything from the log well this hopefully work from all sides therefore if something will get incorrect with your request for the transaction then there is logger and you can list it any time and make a decision according to the log rather than to just depend on the return value from the transaction resultment well what is also different DNF itself has its own configuration but if you use API you will discover that many configurations many API methods that requires parsing configuration directly to these methods the reason is as Paoloa presented difference between C and the Python path configuration is in the Python and the logic is in C and it's not capable to pass the original structure defined in the Python what it means you need to pass less arguments it's more transparent therefore you cannot have a configuration and use different set of the values sometimes you can override it but again it will be transparent therefore you can get the original value that you use for the request well with what we also did we try to listen to you to the community what you need in PAS because many your requests that you know we cannot do it right now in DNF 5 we remember your use cases and try to modify the design to be able to provide or to support additional use cases it means that we try to use less hard coded settings and values in our code and move these values for example for the transaction to the configuration additionally well looking to your code your project we discovered that you have some bundle of our code therefore sometimes make sense to provide it as API then you don't need to for example pass arguments it's like I would like to behave like a DNF but the API or the argument parser is not available therefore let's use the same logic like DNF does then our users will get the same touch like with the DNF the problem is that it's again code bundling and we usually improve this code because we find some edge cases when original parser fail but it's not updated at your side it's hard coded your parser at your application and you have to maintain it and usually it differs well there are of course many improvements in DNF 5 and let me just share few for example performance improvements for example one of the site is loading of the repository with the DNF you can see that it downloads the repository and then it's somehow freeze and then it's continue with another one and that you cannot experience with the DNF 5 the trick is that we split downloading metadata into two processes that run in parallel that runs in parallel it means that when you download the first repository you directly started the downloading the second repository and the first metadata for the first repository gets processed by our sole work this is not only improvement another thing and very important thing is that DNF 5 does not download files by default files are really huge piece of the metadata the biggest one and DNF 5 is not downloading these things it has two sites I mean less downloads less requirements for your hard drive and infrastructure also is used much less and the highlight is that in some cases that I will be not available it means that some packages for example requires some strange files for example guidelines they do have a problem because DNF 5 by default will not see these dependencies as satisfied because the metadata that were used by DNF are simply not available for the DNF 5 the good news is that if your downstream if your third party repository requires such a metadata it's configurable therefore it's not hard coded you see that you end up you need it okay then you will pay the price but don't we should not pay the price for these rare or hopefully rare use cases on all systems oh additionally I have few example of the performance improvements and this is the let's say the long improvements in our DNF stack that well we improve according to the DNF and of course DNF 4 was much faster than DNF 2 and DNF 2 was much faster than DNF 1 and it is much faster than DM in many cases especially for the transaction and we continue with the same same way therefore well as you can see well even repo query if you ask for the multiple arguments or if you run for example what requires for with the repo query or even even update with many arguments there is a huge difference but please take it as an example yeah there are many differences and hopefully you will get satisfied with the result well let me move to our roadmap because I think it's also important where we are we are at the federal 38 I will happen at federal 38 micro DNF was absolutely by DNF 5 no one complains and we received not complaint for this change it means that if you install new containers well you will have a DNF 5 inside well also what will happen well of course the federal that's one important location federal 38 was also a milestone for the DNF 5 project because this is the first time when it appears in federal repositories well upcoming release federal 39 what happen there is a huge change because DNF 5 will absolute DNF what it means by default anyone will start by default using DNF 5 also this is the time where users or users of the command line interface it means other components are supposed to work on the adoption of the DNF 5 it means if your app is using anything from the DNF then well check verify this state before the release of the federal 39 that everything works with default package manager distribution means in DNF 5 if it doesn't well then please report it as soon as possible because there is nothing like unresolved problem you know we are engineers resolving problems it's our job but if you will not report to us what you need then we cannot help you sorry guys the next release Fedora 40 Fedora 40 is the time where we would like to absolute micro DNF or remove micro DNF from Fedora we will set up the standard process you know or retiring the package it's the standard process of Fedora and I'm using this channel just to share it with you because it is important get ready if you depend on something that it's not provided by DNF 5 please don't act no we are taking these requests as a priority it means if any other software is blocked or adoption of the DNF 5 is somehow blocked it's our priority well I've heard several times why we are removing any functional software from our stack and the answer is I remember YAM I remember when and in which state we removed YAM from our distribution well it was removed at Fedora 31 and at that state it was completely broken the problem is that if you will search on the internet what to use with Fedora or REL often you will get a YAM therefore you will install YAM you try to use YAM and then you will get a tracebacks because it was not functional and that's what we want to avoid remove software before it gets broken from the distribution because microDNA is not going to be supported by upstream and the next stack the next milestone is of course the removal of the DNF DNF from the Fedora from the same reason therefore if your application depends on DNF API or LibDNF or Hockey or Hockey API then Fedora life cycle that would be good if you will start or if you will finish adoption of DNF 1 what's your risk standard process in Fedora when you remove the package is well you know we are not going to support it and well with adoption you can take it you can take it you can start with support but you know you have to support that's that therefore and I am passing the line ok so let's see the DNF 5 in action first we have prepared some command line examples comparing the original DNF with the new DNF 5 experience so we have prepared like two separate containers both with the same configuration and on the left side I will show you the command using the original DNF and on the right side I will run the same command using the new DNF 5 so let's start with some common usage like installing a package so we can install for example the Glypsy devil package and the same one with the DNF 5 and what's happening now is that the DNF needs to fetch the metadata from other repositories before being even able to tell that the given package is available so we have in the system configured the links to demo repositories and from there we are downloading the metadata files about what are the packages available, what are their sizes and what's the relationships between the packages and we can see the difference that Iada was talking about when we look at the R38 repository there is like 34 megabytes downloaded for DNF 5 while for the DNF there is like 83 megabytes of metadata downloaded and it's mainly because of that the files are now not downloaded by default and here we can see also the calculated transaction or in other words what's going to happen if a user confirms the installation so DNF tries to find the best available way how to install the package of the given name and if it wouldn't be possible it tries to provide the reasons why it is so we can also see some differences in the output we have some more information regarding the DNF 5 there is like what is being replaced and also the sizes here now refers to the installation sizes of the packages while in the DNF it was the download size so we can just install the packages we can see there are some connectivity issues but there are some changes in the output but hopefully for the better user experience now so when using the command line environment we don't probably want to type a lot so in the DNF there was already some support for auto completion where we for example want to auto complete the mark command we provide with some suggestions there when we press the double tap but if there is some sub command for the given command we want to use the DNF doesn't provide any suggestions and instead it incorrectly provides already the like install packages as the argument for that we can try the same thing for the DNF 5 you can see that mark command is also completed and also the sub commands are provided for the for the user also with some bring description what each command is doing so this is also hopefully for the better experience and now let's try to compare how fast is querying the information about packages so we will first run the most simple query command which is the DNF repo query which basically lists all the available packages from the connected repositories so we can just run it for both commands we could see that there was a little bit faster for the DNF 5 but we can measure it precisely using the time command so we can use the DNF repo query and also we will suppress the output so we will actually really measure the execution time of the command and the same one for DNF 5 and we can we should see there the improvement there is like more than 2 seconds and less than 1 second for the DNF 5 we can also try something a little bit more complex so list all the packages with what depends on the core library and the same thing for the DNF 5 and there the difference should be more significant in this case there is like 6 seconds for the DNF 5 and there should be more than 3 times slower for the DNF yes it's like 18 seconds okay now let's try the same use case as I shown before with the installing of a package but now using the DNF 5 API so this is how a simple python script installing some of your favorite package looks like but let's look at it step by step so first we need to have installed the DNF binding package for python in order to simply import the libdnf5 library like this then we need to create a base object like the center point of the DNF and tell that we want to load the configuration file from the default system config file and apply it then we need to prepare a data about repositories so we prepare a sec with the data about all connected and configure repositories from the system configuration file and then we will download the need in metadata and parse it into the objects now we have like everything prepared to install DNF what we want to do or like what is our goal in the script and here it basically means installing a package of a given name in this case it's the nudoku package it's a simple sudoku game for the CLI environment then when we have the goal specified we can resolve it meaning calculates the whole transaction what's going to be installed, removed from the script which is followed by downloading the needed packages from remote sources and then actually running this transaction which will perform all the needed actions in the system these two actions here are separated because in some use cases we actually don't want to change the system and for example just download the packages right so now we can run the script I will just show you that there is no nudoku command yet and I will just run the API script and it will simply just install the package hopefully maybe and yes it should be now installed and here we can play the sudoku all day long in the CLI okay nice okay so now we are heading to the end of the presentation so I would like to briefly introduce our core team members that are participating in the DNF 5 development so here Marek Boha, David Cantrell, Evan Goudy there are our presenters today Alysha Matej, Jadam Raczek is our great tech lead Jaroslav Roval and Nikolas Sela and I would like to also mention our QE team I think we have a nice and tight cooperation with them and they help us improve the CI stack and implement the API tests for the Python and now I would like to invite you to connect with us and collaborate on our projects our main interface is on the github channel so we have their issues which we would like to invite you to discuss what things are bothering you and what you want to improve the pull requests their review process and etc and also we have we try to make label all the things we think that are good for the new contributors with the good first issue label so watch out for these on the Kevin board you can see all the progress on the work items like what is being reviewed or what is working progress and what's planned to be done soon in the to-do list and for general Q&A there is the discussion session also we are on the bugzilla which we watch this for the Fedora bugs so yeah and we try to watch all the items from these channels regularly every week so we can provide the answer like the response to reporters quite soon okay thank you well the question is that we turn off downloading file list by default but why did not we use file list from the primary is it correct we do we do therefore everything is in primary we read therefore it means if you require something from BTC from the configuration or if you require binary from user bin standard directory it's not from Fedora yes everything will work like charm so the question was when I run whether it downloads file list in background well not yet but for example if you request a file I mean install and any file DNF automatically downloads a file list therefore this already is present and I think we will extend for example if you want to list files and so on that there will be a logic but it will take a time to implement it everywhere I don't know who is there maybe it's Bishak I'm sorry can you share the question you mean hard drive or RAM RAM well the question is well that there are some rumors that DNF5 is using more RAM memory than micro DNF I think this is not true because micro DNF by default downloads also file lists and the most requirements for the RAM is a file list before you process them therefore it means there should be a significant decrease in the default behavior of DNF5 in any operations but you cannot count on that I mean if a user will try for example install file then it will download the file list and again it will require additional memory for processing these metadata well we have no other choice well the question is that there are some rumors that modules are going to disappear and the question is what's the plan for the DNF team well we have no other choice than to implement or modularity use cases because it is expected that on one system you will be able to handle non modular and even modular staff therefore we have no other choice than to implement modularity like it was in DNF last question okay well that's your answer well you can read the history well the question is why not Rust the long answer Rust wasn't an option at the time when we start with the development of DNF5 and we have good reason why to not start short story long story is on the Fedora devil please read and thank you very much and please don't hesitate to ask questions after the session we will be happy to answer any of them yeah okay welcome so yeah I hope you are all here to contribute to Cento-Western Kernel and maybe not maybe you just like get inspiration or maybe get scared away by what we have and what we have built anyway people external to the REL or Red Hat Kernel teams can contribute to the Cento-West stream and by means of REL taking the Cento-West stream to the REL Kernel so first I will spend few words about what is the Kernel package in Cento-West stream it's a package as any other and right except it has some specifics so it's RPM it's actually set of RPMs but the specifics are that the source repository for the Kernel the Git repository where the work is done is not the repository from which the RPM was built this is different to most packages in Cento-West stream so it has separate repository more than later second difference is that the work on the the Cento-Western Kernel is parallel so imagine like bunch of number of surgeons working operating on the same body one of them is like maybe updating the circular system replacing arteries with something better while another one is replacing a leg at the same time so there are some challenges especially when the arteries go to the leg they have to coordinate somehow just to give you an idea how massive it is I checked yesterday so currently we have 149 merge requests opened for Cento-West stream Kernel 9 some of those have several hundreds of commits in them of course to have all of this to be manageable the Kernel teams have to establish some processes on that because it wouldn't work otherwise so there are custom rules custom processes that needs to be adhered to if you want to contribute on the other hand the benefit of all of this is that we basically need a dedicated person to maintain all of that to merge the stuff to take care of building the packages to the disk gate to the repository from which the packages are built so we have a dedicated person or maybe even persons for that and they deal with the Cento-West stream processes themselves so the distribution have some processes but the contributors to the kernel don't have to deal with those so we basically exchange one set of processes, one set of complex processes for another set of complex processes so this is the introduction in order to understand what what the work entails, what are we basically constrained by let's first talk about the expectations so let's take a user, user of Cento-West stream user of RHEL what they expect from the kernel of course stability what does that mean that means no regressions stuff that worked before should work after a kernel update well it's more, it's performance no performance regressions the user probably wants after update for his workload, for their workloads to work as fast as they did before so performance regressions no no API changes of course if you have a program and then update the kernel you still expect your application or program to work basically every application interacts with the kernel so the API that the kernel provides the application should not change at all this is what you expect well then we have kernel modules you may be using third party kernel modules NVIDIA driver for example for your graphic card or whatever and you want those drivers after the update so the kernel update so the problem is that those modules, those kernel modules the kernel drivers, they are using the internal API of the kernel so you want this application binary interface between the modules and the kernel to not change at all well it turns out that some people want more not just the binary interface they want also the internal API which means I take the driver source code from the old version update the kernel and compile it against the new kernel and it should still compile sounds like the same but it's actually a very different thing no behavior changes of course I want the stuff to run exactly as it was running before the update so yeah in short do not touch the stuff I'm using red hat please keep your hands off the stuff and also please bring me new features we want the newest Chinese so please give us what we need there's that new hardware we bought we want it to work in your next release so please update the drivers for us oh and we also heard about this cool new feature that upstream kernel has so please bring it too update the core update everything you can except for the stuff I'm using don't touch that stuff now do you see the problem that my stuff is different thing for different users so everyone want us to update everything except for this stuff so this is the situation we're in we need a compromise of course we cannot deliver all of that at once so the compromise that we arrived to it's a compromise nobody is happy about it but apparently people are still buying rel and using CentOS stream so it's not that bad so the compromise is no functional regression if something worked before it should work after the update performance yeah it's we should not regress performance but imagine cases when you do something and you improve performance for 90% of users buy a lot didn't touch performance for 9% of user and for 1% of users you slightly slowed down their workloads is that okay is it not who knows yeah so there's some kind of balance so yeah but probably slowing down networking for 90% of people are slowing down disk storage that's probably not acceptable then yeah of course the user space API that the kernel provides to the application that should not change but it should not break so we can change it if we maintain backwards compatibility probably or if nobody notices that's okay as well right the kernel modules yeah they should probably work but here is a problem the kernel level of land upstream the community kernel it doesn't care about internal API stability it changes all the time the interfaces inside kernel and to the drivers they are constantly changing so if we are to bring in some new features we cannot keep this interface completely stable we can try to maintain some backwards compatibility but it's like a lot of work and nobody's enjoying that work cost a lot so yeah so we have to limit somewhere again it's about finding a balance we also are updating drivers not all drivers obviously we don't have infinite people working on the kernel but we are updating drivers but again sometimes updating a driver means bringing in a new kernel core feature that the driver is using yeah so there are dependencies and it's not always easy because sometimes bringing in some new feature from a steam it's risky yeah we risk stability we risk introducing regression so again it's kind of balancing act so we might update drivers but maybe not enable all the features that the hardware support supports and of course we are updating the kernel core features again balancing that with stability not everything is possible without compromising the other stuff so yeah as I said I'm always happy but hopefully it's good enough for everyone how does this work in practice when we started and now I've been talking mostly about basically only about CentOS Stream 9 but it applies to also the other streams so for CentOS Stream 9 we had when the development started there was a kernel 5.14 and the CentOS Stream 9 is still on this kernel even those years later so we have kernel 5.14 but we are back porting stuff for upstream we are taking commits from upstream and applying them to the kernel and that ranges from single bug fixes for like simple stuff single features to like large rebases of all drivers and all subsystems so for example for you to know what I'm talking about the XFS file system is up to the upstream kernel 6.0 the USB subsystem all drivers, the core like everything is on par with upstream 6.2 you know the latest kernel in the upstream is 6.3 so this is pretty new in the XFS subsystem this is pretty core stuff that's like connected to memory management and all of that it's 6.2 in the kernel in the current CentOS Stream 9 wireless including all drivers 6.3 multi-party cp 6.4 that kernel is not even yet released upstream so you see all the different stuff is kind of the kernel replaced different version so maybe you get the franca kernel terminology now or I can tell you the term now this is what we have it works it works but one thing I forgot and I will stress it multiple times we are taking stuff for upstream it means everything almost everything we strive for everything that we have in CentOS Stream to be upstream we are taking stuff from upstream everything is to be upstream first so as I said all of this can stick together and work reasonably only thanks to thanks to several things the processes we have around that I will talk about in a minute and the second and very important thing is testing so this is the testing that happens whenever a single change be it a single bug fix or that large rebase whenever it is applied to CentOS Stream kernel first testing happens when a developer does the work so when we back port stop from upstream or when we develop stop upstream and then back for it so this is where the first testing happens obviously nobody wants to just push over the fence untested stuff then automatic testing kicks in we have CKI there where talks about CKI in previous DEVCOMs if you are interested you can find the recordings online so this is continuous kernel integration testing large test suite with a lot of infrastructure around that so this is run we have Ernestiv Linux network stack testing which is specifically for testing networking both functional performance and we are adding more over time so this is run automatically then there is pre-verification which means before the change can be merged to the kernel a human and hopefully or maybe automation if it is there if there is automation for the particular feature or particular bug but someone someone else then the original outer that is important tests this and if it passes only then this is merged to the centoestrian kernel then there is integration testing because it is nice that you test it one feature or one bug in isolation but it can interact with those other 150 parallel changes to the kernel in weird ways so once this is merged we need to test again so it did not conflict or did not influence by other stuff in negative way and last thing is proper QA testing done by QE engineering that is really comprehensive testing of the whole kernel and all the features so that is a lot of testing ok so with that let's let's look how it looks from the point of view of the contributor to centoestrian kernel the centerpiece is the merge request we are using gridlab for centoestrian not just centoestrian kernel but the whole centoestrian development so this is the URL of the centoestrian 9 kernel source repository which is where the development happens it's completely public you can go there you can watch what we are working on and of course contribute so merge request against this repository that's basically the centerpiece around which everything is built however before you can open a merge request to file a bug or issue every change we do needs to be tracked so you go to bugzilla or jr doesn't really matter file a bug or open an issue there be sure to select the proper product component subcomponent product and past linux 9 component obviously kernel subcomponent in bugzilla whatever you think is the best fit the reason for that is you want your change, your merge request to reach the proper team in the bug or in the issue explain why you are doing that explain the benefits of it why it's important if you come as a contributor to centOS team kernel it's not the amount of work that is connected to the contribution it doesn't end with submitting pages, submitting comments there's more, there's testing I talked about someone has to do it and there's maintenance, someone needs to maintain that stuff for the whole period where red enterprise linux is maintained which is many years so submissions don't come for free for redhead so explain the benefits, explain why the change is good and also because it means work for other people try to make it easier for them provide detailed testing steps so the quality engineering is okay with the change because they know okay this won't be that much work for us don't forget to note that you will be submitting the main request so you are not asking the developer to do the work make it easier for them and be mentally prepared for the option that it might be rejected so we have a bug or issue then now let's do the work we clone the repository as usual from gridlab starting create our own branch based on the main branch this is where the main branch is where the development happens apply our patches and now let's stop at that now this is the first thing about the processes the commit messages they have some mandatory format I will explain it on the next slide so it's not like freeform commit message so that's the first thing apply upstream commits one to one so one upstream commit is one CentOS stream commit do not squash commits together and of course it's a Frankenkernel so it's unlikely that commit from upstream will apply as is to CentOS stream kernel most likely some commits do if you're lucky but most likely you will have to do some changes adjust to the old APIs that are there or something those changes need to be explained there should be reasons why why the the back port differs from the upstream let's look at the example this is a real commit from CentOS stream 9 and let's see first the mandatory fields there must be a bugzilla or JIRA link in every commit every single commit must have this line and in this format it's parsed by machines so even like training white space is a problem you will be taught so it's not like it's not entirely black box more on that later so this is the first thing second thing every as I said there is upstream first policy so everything that we bring to CentOS stream kernel must be upstream this line actually says this is the hash of the upstream commit this one is back porting exactly this format it is parsed by machines third thing that's mandatory is the signed off byline which has some legal implication if you're not familiar with the certificate of origin see the documentation must be there and here I will point out this section this is not parsed by machine this is for human consumption for reviewers except the conflicts line that must be in this form explains what was changed compared to upstream so this is an example how it looks like so you apply patches and then you can finally submit a merge request the usual GitLab workflow nothing you are not familiar with so basically you fork add remote push to your remote open merge request using web UI or API and you're done almost in the merge request description you have to repeat two things the bugzilla or gilaring and your signed off byline when you do it then the information kicks in on GitLab kernel is built and it's tested if you are an external contributor then only limited tests are run it's not full test it's for security reasons once your submission is reviewed by someone from Red Hat they will start the complete pipeline and we don't want the code compromised by external contributions obviously then also some automation is run which will do checks on the submission it will check better the fields I described like all that bugzilla and so on it's present it will tell you in a comment in the merge request it will check whether the bugzilla or gira issue files whether it's in proper state there's something you cannot influence someone in Red Hat has to put that in the correct state the bug or issue another check is for missing fixes so if you back ported something from upstream and there was a follow up upstream that fixed that it's not there you will get notified it will also check for conflicts between submitted merge requests it's quite important given the amount of merge request we have then there will be some labels added actually quite a lot of labels so this is an example of one merge request there are subsystem labels here basically those indicate which subsystem were touched or were modified by your submission this is a large merge request it has I don't know like three or four hundred commits so it touches a lot of subsystems there are labels with the current state like exnet needs review or CKI and so on it will also add approvals to to get lab to merge request meaning list people who should review the merge request and it will notify them then review happens this is important part but it's nothing surprising so someone from Red Hat or from community will review the commits to submit it will make comments now it's important to respond to the comments if someone has objections or ask for explanation provide that the discussion happens in the merge request in the comments there and when finally when it's reviewed the merge request is ready for QE testing that means you get all of these labels in the okay or warning state at worst your commit description is okay no merge conflicts the CI testing succeeded and somebody or all the approvals were given all the needed reviews were done then and bugzilla and JIRA is like in prepared for testing state you get ready for QA then the testing happens the pre-verification again done by someone else then you you will probably agree in the bugzilla or JIRA who will do the testing whether it will be Red Hat or whether someone from your company or your friends provides the testing whatever then if it's done then you get bugzilla or JIRA okay ready for merge it's merged or the maintainer will merge it one short notice sometimes if the for example if there is a driver update it might depend on other changes in the kernel so if you for example update the networking driver your favorite networking driver to the newest version it might need some changes in networking core to support new protocol perhaps so that's a different merge request of course it's not the same stuff so different merge request needs to be filed for that and to save us time you can actually submit those merge requests in parallel you can depend on other merge request which is not yet merged by pulling the code applying your purchase on top of that the only thing you have to do is put this line to the merge request description that you are depending on this particle another merge request and then the automation we have will be able to figure it out find out that what is the first commit that is really part of your merge request so that's like top of the dependencies will put this label there and then you're good the lab is not aware of that this is a feature that is missing in the lab so we implemented it ourselves and as a consequence it's really hard to impossible to use the web UI I will ok 15 seconds we have tools that will help you with that the lab is a CLI tool that helps with submitting merge requests reviewmatic which is a custom tool that helps with reviews that understands the web flow we have so that's it here's a link for full documentation web process if you're interested in details and now questions so the question was that by doing the back ports we are repeating the work that upstream does with the LTS kernels and better it would not be easier to use them yeah the problem is that the quality of the upstream stable kernels is not satisfactory or a distance we found there are things like machine learning involved which selects which commits are back ported and also basically whatever someone thinks is decidable for those stable kernels strewn over the wall and applied and it does not mean this does not meet the stability and no regression guarantees we need so we really want all the stuff that's going into that to get reviewed by us and to be tracked to have someone who's responsible that's like we can go back to say ok yeah you broke it you fix it yeah so all of it together you found out it's like for stability it's basically no go yeah there's question yeah so the question was what what the integration testing consists of what it is basically right now it's done by CKI again but it's like larger battery of tests a lot of different tests I don't remember but there's LTP and like many more the question was whether they are open source I believe so I'm not 100% sure of everything but I think so please don't like yeah I'm not sure I'm not sure thank you hello everybody have you ever wondered how to maintain an RPM package in Federa, CentOS, CentOStreem, REL or any other if so I will try to answer some of your questions today I'm a little bit afraid that I will open more and more questions than I actually answer but I have plenty of time after the talk so we can discuss it my name is Lumier I'm from Python maintenance team in Redhead and we maintain Python packages quite a lot of them without further ado the agenda is kind of packed I'm not sure I will manage it in 25 minutes so I will try to do it quickly but we have plenty of time after that for questions so let's start this is probably well known Bell Curve describing the normal distribution of basically anything and if we have it for the users of a software then the average user is somewhere in the middle and the line here is a software and on the right side from your point of view it's too new not well-adopted, not tested and so on and on the left side is too old absolutely full of flaws and full of problems and security bugs and so on and it's average so my wild guess is that a lot of you is somewhere here in the middle something which is still supported but well tested, not too new not too old and that's fine the problem for us especially in Federa and Redhead Enterprise Linux is that we are on the both ends of the both extreme no software is too new for us I will show you some examples and no softwares are too for us, I will also share some examples of that basically the flow of the code looks something like this that's if you really don't know what does it mean to maintain an RPM package that it might simplify it look like this so somebody creates a software, library application whatever and on the other hand there are users who want to use those applications and libraries and in the middle is a magic man or woman the maintainer, package maintainer because usually if you want to install some software use it, you see it on the internet and oh, this is a very nice application and you really want to install it if you use a Linux distribution probably your first step will be dnf, search yum, search, epithes, search something, something and you really wish the application to be packaged into that distribution because it means that you can install it and somebody else take the care of all the stuff around it and compilation and testing and all stuff and that's who we are in the Python maintenance but not only in that one team also in other teams as well we are taking the code from upstream creating RPM packaging and delivers those packages into upstream into our users, customers and so on and so forth I won't go much into detail today I won't be telling you about the comments and pushes and PRs and updates and here and there what I would like to show you is what the challenge is you might expect if you want to be RPM package maintainer so I will use a couple of terms here the list is short when I say upstream development I mean the development on GitHub, GitHub somewhere the upstream code applications or libraries when I say downstream development I mean the development in the Linux distribution the packages who are packaging the stuff writing spec files and so on when I say CVE I usually means some security flaw security vulnerability and EOL is a shortcut for end of life software without any more upstream support so let's take a look if you want to be a package maintainer what's your responsibilities you should try at least your best to keep your package functioning which usually means that it should work of course it should be buildable from sources so the process you describe how to build it, how to compile it how to test it and how to create a package from it should be repeatable after some time it should be buildable from sources it should be installable the fact that you have RPM package doesn't mean that it will work on your machine so it should be installable and should just work it should be up to date so we should update it regularly you should fix issues and potentially CVEs that's something I will describe later in the deeper detail you should also try to limit the impact on other packages which is something we will also focus a little bit later and you should try to limit the impact on the users which is even more even harder than limiting the impact on other packages but the best thing you can do as a RPM package maintainer is that you stay invisible if everything works nobody knows about you because after you install it and it works I don't care who maintains it but if something breaks then it's your responsibility let's start with some very very simple example the package name IOPING that's a very simple utility it can measure the latency of your hard drive the same way as the classic pink measure the latency of the network that's it in C or C++ if I'm not mistaken and it needs only two dependencies to build GCC and make use make files and it needs to be compiled GCC and make and only one dependency to run which is the core of all the Linux distributions so nothing you should be worried about and no packages needs IOPING to build and no packages IOPING to run which means that this package is a so-called leaf package if you imagine the tree of dependencies of RPM packages that one would be at the very bottom nothing depends on it which makes the situation for you kinda easy for the maintenance because the probability something breaks for you is very low well if anybody moves the GCC from 15 to 20 then it might break for you but other than that the probability that something breaks for you is limited because you have very limited set of the dependencies and the probability that you will break something for anybody else is also kinda limited because no other packages depends on those on that package directly so only if you break it you might expect some bug reports from users and that's something which is really hard to measure because we don't know how many users are using the specific components the specific packages so what could well wrong me as I mentioned somebody can find a bug in the application that might happen of course or security vulnerability and the problem might be upstream so in the code of the IOPING itself or the problem might be downstream or in the files or manual pages or documentation whatever or the problem might be on the left side of the previous image so somebody update or change something and it might break your package or you change or update something and it might break other packages or the expectations of users in that case what to do as an RPM packageer well if the problem is upstream test it in Federer and propose the fixed upstream and we are doing it a lot that should break the first kind of expectation or that basically RPM maintenance just taking the code packaging into RPMs and giving it to users but that's not usually true and I will describe it a little bit more detail later the problem is that in Federer we use the latest greatest software and that means that we are testing the software with the latest possible components of the dependencies and that means that it usually breaks something especially in case of Python so we are the first ones who actually see the problem so it's not just about hey I can report the problem which is the second best thing you can do I can just the report it and wait for the upstream to fix it and after that happens I can just take the patch or the new release update in and it's done usually it's not so the best way you can help as a package maintainer is prepare a fix you can test it on Federer where the original problem appeared so basically the environment you want to test it is actually already prepared for you the second best thing is to understand the package and you can provide all the necessary details all the versions of dependencies and configuration flags and compilation flags and all the details around that might be really helpful for upstream and by doing that you might actually make upstream to fix that problem very early for others so you uncover a problem you fix it, report it and those might make other users on the bell curve in the middle will ever know about the problem being there at all or you can do nothing which is bad if you have broken dependencies then the first point is the same as a bug or security vulnerability you just need to take your software and adapt it to the newer version of your build or test or runtime dependencies which might be a little bit hard if you are really lucky and somebody is moving a package to another my version like switching from gcc10 to gcc20 for example if you are lucky they are aware of you depending on the gcc10 and they might create a compact package for you which means that you can switch the build dependency and say ok for now I'm fine I have to take a look on that because the gcc10 won't be there for a long time but right now I'm safe users can still use the latest version of ioping but I have to think about it but honestly that doesn't happen so often so you have to switch your package to the latest version of dependencies and of course if the problem is downstream in your package then it's your responsibility right that might sound like a lot of work does it that's nothing not yet we are not there yet let's move to the other side of the problem because we have a cookies not actually cookies but we have a lot of stickers so after the dog I will give you some stickers so let me step aside for a while I will describe in a more detail what we do in Python maintenance especially in Federa and it loves Python almost all of them Python 2.7 not that much but still and we would like the Federa to be the best distribution for Federa developers at all and for Python developers at all which means that we maintain multiple Python interpreters at the same time continuously for some of them mainly because it's the same in REL and that means that if you want to develop a Python application for REL 8 or for REL 7 or for REL 6 or REL 9 and future 10 then you can use Federa for that you don't need a virtual machine or anything virtual with all distributions you can take Federa take the Python whatever version you want you can create a virtual environment and it's all great if you are on the other side of the bell curve as we are sometimes then you might want to test your application with the latest, greatest version of Python which is also possible we have Python 3.12 which is now first beta but we have it in Federa since first alpha which means that you can really test with all the supported Python and we support a lot of them much longer than upstream and also we have some alternative interpreters like PyPy we had also Jyton but not anymore and also we are very fast really it took only 7 days for 3.11 alpha one to appear as a build in Koji which means that if you really need the latest, greatest Python there is absolutely no need to compile it from sources from you and that's really, really good first beta and only 3 days for 3.12 alpha one after it was released upstream it appeared in the build system of Federa and you can use that which is quite awesome and what's maybe more important than that is that these are much faster than the usual CI providers you know that you can use Python for testing on GitHub actions and Trevys and CircleCI and whatever but you will wait months after the official release of Python 3.11 or 3.12 in the future you will have to wait months after the official final release to appear in the CI system like those you are not a Federa user shame on you but it's not a problem because we offer all the Python including with the TOX which enables you to test one application with multiple Python versions all the Python TOX based on Federa as a container image available on Docker Hub and also as a GitHub action so we don't need to use Federa directly you can use it only in the CI to test with all the supported Python versions alright but the situation for most of the interpreters I mentioned on the previous slides are the same as for the IO-Pink because the alternative versions are basically the leaf packages as well as the IO-Pink because nothing should really depend on it they are not meant to be used in production they are meant for development and testing but there is one the main one the main troublemaker and there is only one in every Federa release which is the main one it's basically usually the latest stable release of Python and the next one which causes us the more troubles the most troubles and most work at the very beginning of the Federa release because all of those are usually leaf packages for testing and development that's fine that applies also for the future versions but the main one is the troublemaker here let's increase the complexity of the example let's do the IO-Pink a little bit the graph for Python 3.11 which is the main Python interpreter in Federa 38 and Federa 37 looks like this we need 45 packages to build it we need 24 packages to use it and by use it I mean the Python interpreter and its standard library not all the components provided by that package but that's the part we really cannot affect something might break here and the probability on the left side of the image is much higher than for IO-Pink because those packages are much more complex than GCCN make and the numbers are just really, really higher but the funny part begins on the right side 4,400 packages need Python to build and 5,600 packages need Python to run like wow so what we can do we can bump a version building in Koji and release and do whatever you have to fix it for yourself let's say that's obviously something we cannot do and that's the reason why it generates so much work for us the reasons are basically two the first one the newest Python releases or every Python release usually contains some backward incompatible changes like removed the deprecated modules because when developers see deprecation warning for two years or three years that's not enough so when the module disappears from the standard library then a lot of packages or a lot of tests and stuff like that starts to fail, right? so there are some backward incompatible changes and also one more problem is that Python is interpreted language and that's true but a little bit of that is compiled let's say and that compilation process creates the PYC files which is basically a cache version of imported modules the problem is that the imported modules the PYC files are written under hard drives to the locations where only root or super user can access which means that we have to we have to ship those files in RPM packages and that means that those are installable only by root so if a user imports a module and it tries the Python tries to create the PYC files somewhere it's not possible then it won't happen, right? and the compilation step would be doing the same again and that means that we need something called mass rebuild but that's somewhere in the middle of the update process the first step of the update process is write a change proposal you can see those on the Matthew's talk today so write a change proposal describe your plan describe the contingency plan and so on and so forth then package a new Python version as a new package we cannot just update the old one we want to maintain it so we want to keep the old version so we have to create a new package and then rebuild the thousands of packages in copper copper is just let's say an alternative build system we can use for testing and during those rebuilds you will uncover a lot of bugs some of them you can fix some of them you cannot you have to report the bugs upstream of the libraries Python itself because we are just humans and Python is just a software full of bugs as well or downstream fix it downstream in the Python packaging as well so really incredible amount of work and it usually the whole process I'm describing here which usually takes the whole year because Python releases are somehow aligned with the Federa releases which means that fall this year we expect Federa 39 313 alpha 1 and we will have the whole year to prepare the Federa packages for the new Python version rebuild all the packages in the SciTech again the SciTech is something like a branch in Git nothing to be really worried about so we prepare all the packages we know about all the problems we try to fix in copper and then we can rebuild them again in the SciTech of the development version of Federa actually happening right now the process of rebuild in the SciTech started a couple of days ago and we are somewhere around 3,000 packages already built and then you merge those rebuild packages back to raw height and ta-da you have the Python in that case 312 in raw height and that's awesome it's incredible amount of work and thanks mostly to Tomasz you can talk to him after that Tomasz is doing incredible amount of that work for through the whole year maintaining that process it's incredible and some numbers 3,361 packages in copper almost 50,000 of builds and just almost half a thousand bugzilla and those are just bugs in bugzilla I'm not talking about all the GitHub issues and so forth so in Federa updates are really complex and you have to be really careful because you can break all other packages or a lot of system tools which is case for Python because a lot of things really in the system depends on the Python itself like the installer Anaconda and the DNF version 4 the 5 is not longer in Python but the older version is so you have to be really careful but the benefit for the updates is that it usually fixes the security flaws for you and the bugs so if you cooperate with upstream it usually don't take that long to release a new version of the upstream project and you can update in Federa you don't have to make your own fixes you can but you don't have to and you can do it in a similar way as we do you can assess the impact of the change in Federa if you want to let's stay here in this room and Karolina will tell you more about how we do it in a copper in a little bit more detail but let's move to the other side of the bell curve what if we cannot update what if we cannot change it that much that's a problem we have to solve in a rail the promise for rail customers and users is that we will try I added the dryer mentioned we'll keep your system secure without breaking bad bar compatibility of a provided components which means that the updates are usually out of question usually not every time and I would like to describe that on two security vulnerabilities we had to fix and how we deal with that on multiple levels to show you that it might be really interesting and also really complex fixing something in the older components in the old systems for the customers who depend on the maximum stability let's take the first one which is a web cache poisoning in your lab which is a standard library of Python and the problem is that when you have a URL address here it's usually parsed somehow by a proxy and you have to set the delimiter for the key value pairs here and Word by Web Consortium recommends using the ampersand only but the Python had semicolon and ampersand both at the same time as a default all right we don't need to go much deeper into the detail the thing is that if the application is configured differently than the proxy in front of that application it might mess up the results in the proxy and that's what the vulnerability is called web cache poisoning okay so what we can do we can follow the same recommendation and that's exactly what upstream did all right recommendation is to use ampersand well switch the default to the ampersand that's too rash that's too rough find somewhere middle ground all right let's switch the default to the ampersand and allow users to configure something else that sounds great but the separator keyword argument added into Python don't allow you to use more than one character so you have to choose either the ampersand or the semicolon or whatever you want to but not multiple at the same time and that's a problem right so the Federa we are latest, greatest, whatever it means so we just followed the upstream resolution we updated all of our Python's because the fix was kinda quick so it was implemented upstream they did security releases so we just updated all our components and for the AOL interpreters which are maintained by us and not upstream anymore we just backported the patch so Federa follows upstream so default is the standard and that's what it is, deal with that but for the rail we cannot do that that's a problem so we basically took the patch from upstream adapted in a way to allow the old behavior by default and added the warning so if you use that default so which means that your application is not ready for the new codebase if you use that default somewhere in your rockwheel appear a warning link to the documentation and the documentation will describe to you what's happening but after that we still cannot expect our users to change their codebase so we have to find a way how they can set the new default without actually changing their applications so in the patch in the same thing we have provided a way in the Python code itself which is kinda obvious but we also provided a way how to send it, set it we are a configuration file in ETC and also a configuration we are environment viable so we didn't change the default behavior, we added a warning to our customers saying don't do that please because on the other hand they might use the default for Python and then might have some old proxy configured in the same way and everything is fine if we change the default it might silently break a lot of stuff silently, that's a problem so we added a warning we added the possibilities how to configure that so everybody can do that on any system setting variable adding a file into HCC and whatever so we had to be very careful and in that specific case we have decided to not change the default setting another CVE and the last one that we have talked today about is the tar file module directory travel zone and if you know that the second part of the identifier is a year then this one is waiting for fix for a long time really I've tried to set to Peter like hey we can wait a couple more years and then we can celebrate the 20 years of that CVE and fix it right that would be great birthday gift but no Peter decided to fix it this year and we are actually in the process of doing that so I will describe that so the problem of a tar is that it was designed to back up the whole systems complete, completely which means that it supports Simlink, Hardlink Extended Metadata some special files relative paths and a lot of stuff the problem is that if you unpack an archive from untrusted source it can completely break her system so it is not something we would usually need these days we are no longer back-upping our system on the tapes so that was it but then there is a problem what's the correct way how to do that and the upstream had the perfect resolution to nothing we said in the documentation that blindly extracting archives is a bad idea you shouldn't do that and you should inspect the archive before you extract it blindly alright that's true the documentation said that but the problem is that still the best place to fix it is in the tar file model the best I'm going to be pushed from the stage no no two more minutes please the best thing or best place to fix that problem is really into python even the documentation says that everything is correct in the python so peter by another colleague decided to change it into python but it cannot be done in a backward incompatible way he wrote a PEP 706 which is python enhancement prokhozer really low document full of useful information if you want to read that and it added filters to the tar fully trusted which means that you know you have to bring tar somewhere in between and data which is the safest one only allowing some capabilities of tar python now have a deprecation period in python 312 and 313 the default is the same but there is a warning if you use that default same thing as the neural lib and then in 314 the data will be the new default which is how it works in python so in federa we basically followed the same thing and backported the patches into the LL interpreters so now all of them behave the same way and in rel we have decided to do even more strict thing and we backported the patch and set the strictest filter to be the default one in the rel distribution so we can finally say that after many years the vulnerability is finally fixed and that's one backward incompatible change we had to do and we will see what we will cause so some conclusion some conclusion being a package maintainer might mean a lot of things a lot of different things for a lot of different people if you want to maintain a leaf package with a useful utility please do that it won't cause the whole free time you have but if you want to maintain something like python in federa rel be prepared to sacrifice something and in federa it's good to know your packages because if you do so you can prepare patches you can help upstream fix the problem if upstream is no longer interested in their packages you can maintain them you can help to evolve the whole ecosystem or at least create reasonable reports in rel you have to you really have to understand the packages you have you really have to be able to backport patches just down the road to the very ancient component open the book full of dust and backport the patch into that so you have to really know what you are doing and you have to be prepared to do some tough decisions like we did when fixing the vulnerabilities alright and that's all thank you for your intention because I repeat the question why we have in copper less packages actually depend on the python that's a great question you actually look at the presentation that's great so the reason is that we follow the dependencies between those packages and we don't try to blindly build everything we first prepare the dependencies for all the packages and then we add them so it means that we weren't able to build all of them in copper but the side tech we built which means that some of the packages were in the buildable state so we didn't add them to the copper because not enough dependencies at that time thank you for the question well the question is whether we try to discover runtime issues with the new python that's a great question and honestly the amount of work we have with the build time is a lot so there is not that much time to do so but we really think that every single rpn package should contain tests so we really depend on the components depending on the python to have some tests which kind of means that we should be able to get them to get the problems in the runtime but if the package has no test there is no way to verify it works with the new python yes, unit test mostly because during the build in the build system there is no access to the internet so no that much of integration testing and stuff but every single package should have a check section with the unit tests and if we rebuild the python and then rebuild the py test we hope that py test has as good coverage as possible so it will uncover some possible problems and I'm sorry for that thank you very much hello everyone my name is Karolina Sorma I work at redhead as package maintainer I do a lot of package updates in my daily work most notably python Sphinx which is quite popular documentation tool which has got over 1000 dependencies and every time I want to bring a new update of python Sphinx to Fedora it means that I'd like to be pretty sure that this update will not break anything Lumir has provided a lot of context just in the previous talk why it's important so if you haven't been here make sure to check out later the Lumir stock the process I'm using in my daily work I'd like to describe here today for you this talk is going to present a very practical approach I'm not going anywhere deep in my explanations there is no time for that it's going to be applicable immediately for python packages and for packages that are built from our languages maybe with some tweaks I'm not very familiar with anything else but we can talk about it later in the social event the projected audience for this talk is me 2 years ago when I started my journey as a package maintainer I really needed a lot of context in order to be able to do my job so how does this typically look like there is an upstream project developer someone who creates a really nice library they create a new version me as a package maintainer somehow get to know that there is a new version available that it will be really nice to bring it to Fedora Rohite because this is where the newest freshest, hottest things should go to it's the developmental version of Fedora really good place to learn such updates and that's it I package it, I push it to Rohite build it, I'm done well this is the place where a decision can be made I can either do that and do it just like push it to Rohite I build it locally, it builds great or I can just step for a moment and think maybe I could leverage Fedora quality tools that are available in the ecosystem and what would be that quality tools in a very simplified manner and please don't quote me on that because I have no idea how the Linux distribution is built really distribution like Fedora consists of many RPM packages and in the ideal state those packages are buildable and installable unfortunately that's not the case for all of them but we aim for that and in order to ensure that the update I'm bringing to Fedora is buildable and installable I've got some local tools that I can use which is mainly mock or I can build my update in container or I can leverage Fedora CI Fedora CI does scratch builds and any tests that you have with your package in the Fedora disk it and there is Zool CI which does everything Fedora CI does plus much more so if you're going to leverage Fedora CI go for Zool CI with the only disadvantage that it will take some more of your time and it would be best if you just opened a PR pull request to your own package in order to get any failures before you bring them to the distribution but it's not only about the particular RPM packages when we create the distribution the whole point of creating a downstream distribution is that the packages actually communicate with each other they play with each other nicely which means I can build my package on top of the other Fedora packages and I can use my package to build some other Fedora packages also again Umer has got a great graph showing exactly that and for that for checking the integration we've got a process we called impact check in copper language it's called mass rebuild and you can think of it as an integration test pretty much while the CI is more of a unit test it takes care of the health of one package impact check makes sure that no other package stops being buildable when you bring your update to Fedora Rohite in copper language copper is a build system it builds packages its main usage is not how we are using it we are creating a lot of copper repositories with pro-oA builds and in copper manner what I want what I really want to have is this last build status succeeded this is the only thing I care so the process looks like that I create a new isolated Fedora virtual Fedora I bring my just updated package to that copper repository build it and then take all of the other packages that depend on my package and build them if they all build I'm happy probably my update doesn't break anything if there is something that failed and I really don't like that red journal there then I should investigate and take a look at it so in order to show you this process I took python rich library it's a really nice library to build text user interfaces and the authors of that library and the maintainers of that library in Fedora didn't do me anything bad so they don't deserve what I did to their package I just removed the emoji module I just prepared a downstream only patch which removes one of the modules the package has it's a silly update it's just something I did downstream the upstream developers wouldn't probably agree with me but it's not so far from the truth many times it happens that when you have a major version coming from upstream it contains a lot of removals or renames and some incompatible changes so it can and it does happen and I can still have some packages in Fedora that rely on this emoji module to be available to build and to run so I'd like to know whether there are any and which one and what happens there I've got 32 packages that require python free rich on build time and 22 packages requiring python free rich on runtime copper is a build system I don't have any means to check the runtime thing unless those packages have upstream tests if they run the upstream tests they should pull the runtime dependencies in the build time and then that would increase the visibility for me so the question I'm left with is how many of those packages those build time those packages that have python rich on build time will just stop to build so my catastrophe that can happen is 32 failed packages to start I assume that each one of you has already done the Fedora packaging tutorial and has got installed the basic packages in this process I will describe I will use tools that come from Fedora repository some additional ones Fedora package contains pkg name very nice tool to get just the package name from the whole rpm name string Fedora repository contains repositories I can query I don't want to install packages to my system from Rohite wouldn't be a good idea but I want to query through the RPMs and the metadata there is a nice binary copper CLI to enable me interaction through the command line so that I don't have to click on the interface of copper and more useless parallel comes with a binary parallel to speed things up if I want to send 1000 builds to copper I don't want to wait consecutively one after one the really great tool that comes with dnf is repo query it's complex and it can do a lot of things and it probably deserves its own talk so I'm not going to go into the repo query and what it can do the basic commands to query the repositories is just as follow with the first command you will list all the RPMs all the build RPMs that are in the Rohite repository and all the source RPMs that are in Rohite dash source repository together dash queue and one option to surprise the metadata check which is not something I'm interested in so all the commands have this optional dash queue for a released Fedora the query is a bit different so I just put it there for documentation purposes but I will only continue with Rohite so first question I need to know what gets built from my package I already told you that I just destroyed my package but I'd like to know what is the build RPM what is the name the other packages will depend on in Python case the convention is that the source RPM are not versioned this is the name of my Fedora component but what gets built is Python free dash reach and this is the name I'm going to use in the following commands there can be multiple built RPMs coming from a single RPM and in such case if you'd like to do an impact check you should probably repeat all the following commands for each of the built RPMs that come from your source RPM any other packages can depend on any of the built RPMs that come from yours I will not begin to that too much build time and run time dependencies for the Python world the same invocation of Python reach requires in build time I need to query repository with the option dash dash requires and provided the source RPM name there can be the full name of the RPM and what I get is the full list of built time dependencies if I'm interested in the run time dependencies which for the purposes of the stock I'm not I would invoke the same command with the with the name of the built RPM it's significantly smaller but also I can see that the run time dependencies are listed in the build time dependencies so my package pulls them and probably tests with them so I'm fine here back to the build time dependencies I want to know what other packages are my library Python reach to build I already told you that there are the answer to life universe and everything is 32 but how did I know that there is the commands of course like for everything the repo query that will tell me there is the option dash dash what requires which takes my built RPM as an argument recursively so that I can get the full picture of what would get broken if I really push to the update to the repositories and I'm interested in only in the source RPMs which are the ones that expose built time dependencies those are the ones I can check in copper from all of the results I will pipe them to the pkg name too and this will tell me only the package names so if I run the same command you can see that it gives me something and if I make my bash count that is 32 this is what I want to build on top of my update in order to assess the sanity of my changes knowing that I can go to copper anyone can use copper you only need to have fast account, feather icon system and API token which you can easily get from the copper feather infracult.org api page once you're set we need to do those five steps I mentioned at the beginning so we need to create a new copper repository add and build our package with the update add and build all the other packages first of all all of those steps just before the talk so that I don't clutter our time typing those beautiful commands in my terminal but we will go to the live demo with the results so first of all I do I define a handy variable only so that I can reuse the commands every time I do a new impact check and I do them a lot creating a new repository means that I need to invoke copper binary with create argument it takes another argument name of the copper I want to create I want to create it for Rohite I don't care about architecture really because Python packages are mostly no arc but it may be different for your case there are different routes there are also not only you can also build RPMs for other systems not only for Fedora I'm using copper so it's really universal the important thing is that I'm adding a new repository to my copper it's not only looking at Fedora Rohite but also to Koji why is that? there is quite often a little bit difference between what's in the Rohite repo and what's in Koji packages change all the time the state in which the repository will be 20 minutes later will be completely different sometimes it's like some changes sit in the Fedora repository but the maintainer deliberately or by mistake forgot to build it so we are grabbing the latest packages the latest versions from Koji in order to test on the and because this is a pro way repository I'm not going to keep it forever for small updates 14 days is just enough so I'm deleting after 14 days only big updates require more time adding the package I already conveniently destroyed my package and sent it to my fork of Fedora this git so it's there sitting in the branch DevCon that package I can tell Copper that I want to edit from the source control management system so for that I have an argument at package SCM and again to which Copper what's the URL where my package sits what is the branch name where I pushed the update and last but not least the name of the package I want to build which is Python Rich in my case I didn't have to go to my fork of the package I could only build my local source RPM but then I would not in order it's easier for me to switch any automation I would like in order if I wanted to tweak with my package and push new changes to my fork then Copper has the ability to pick them automatically and when I have added my package to Copper I should build it for that I have a command Copper build package where and what and now I should wait this is important step I should wait for my package to build successfully and only start pushing new packages for that when it's ready if I do it too rapidly and the package will not the Copper will have no problem with going for my Python Rich from Koji repository or Rohite repository and I will not test at all what I wanted to test so provided that my package actually built and it built successfully if it didn't build successfully then I need to repeat it so long as it actually builds otherwise I will not test anything again I can use the utility code parallel to speed things up and another great feature of Copper is that it can allow me to add packages directly from this kit and if I don't specify which this kit it means Fedora this kit so I'm adding packages that are sitting in Fedora this kit all those 32 packages to my previously defined Copper I want to switch on the webhook rebuild it's really nice piece of automation I just mentioned before I didn't call it like that anytime a package maintainer any other package maintainer does a change to their package while I'm doing the impact check while my repository is on in Copper the Copper will automatically know that there was a change of state in the Rohite repository and will automatically rebuild the newer version for me so I don't have to think that oh I did my impact check two weeks ago and it's probably still now because the packages move so fast my Copper will pick it up and another argument dash dash name another option I need to provide to the command all of the all of the packages that require my package build time which is basically again the output of the command you've already seen piped to sort and unique just to be sure that there are no duplicates there shouldn't be but it doesn't hurt last step of the process building the depending packages again using parallel to speed things up I can send to parallel the command build package and it has it looks very similar with some new options so no weight dash dash no weight is an option not to clutter my terminal output I will not have to wait for each build to conclude in order to another one to be sent and the factor of niceness background Copper has got different queues there is a normal queue for normal packages and there is a queue that gets populated in the background so when someone has got a priority to work they should probably use the main queue but impact checks are rarely that important so it's only nice for the other users of Copper not to clutter all of the available workers with my 1000 packages that I will just remove when I see that they build successfully so background is really nice option please use it and again I need to tell each packages I want to build so this is my magical invocation when this all happened my biggest question is what failed this is what I really want to know maybe nothing that would be great maybe something that's not so great with Copper I've got another handy API to interact there is a monitor each Copper has got information about its state how did it go if I just run Copper monitor like that the output will be provided in JSON which is not so nice to interact with from my command line so I can tell that I want my report as a text row and I only care about failed state whether the build succeeded and the name of the package that succeeded or not and in this case I'm only interested in what failed if I do that I hope it will work yes there are 4 packages that failed in my Copper that's not too bad I could go through the build logs of all 4 of them but it can easily happen that I have 10 or 20 or 60 packages that failed my Python interpreter colleagues have thousands of packages that failed it's not that nice to go through the build logs what we often do we use that command to create another control Copper and I have the habit of naming it completely the same with the dash control in order to speed up my typing Kung Fu so with that control Copper I would rebuild only the packages that failed in the first Copper to look for false negatives this could look pretty much like that so provided that I got one Copper repository with failed packages I created a new one control repository which rebuild only those failed packages and from those I'm only interested in those packages that actually succeeded this is my difference what failed in the first one and what succeeded in the second one if the list is long then I can use another Linux utility to compare both results in my case it will not be that bad because I've already done my Copper control just before the talk and now I see that there is only one package out of those 4 that actually succeeded in the Copper which did not contain my update this is my problem this is my Copper this is something I need to really look at that's the process and now we come to results evaluation and this is for now a manual step all of the steps I showed you can be somehow automated you can write Bash scripts you can even script opening a browser with logs from the command line it all depends on the workflow which you are used to each one of us in our team does it a little bit differently so we don't even have one process for all of us but we have automation to some degree in my case I will just go to Copper for the moment I'll show you the most mundane thing that can happen so I'm in my DevCon Prep which is the repository I was doing all the work before the talk I will find packages it was Python Typer that I'm interested in it actually failed not that I didn't believe the CLI but it did there's a lot of builds and they all failed that's not a good sign it's not probably just a flaky package you can see that my build was running in the background and there are the logs there is the Groot, this is the Rohite build which failed and there are the build logs that I'm interested in what gets open is a really nice build log everyone who built a package knows them by heart and I'm of course interested in whatever is at the end of the build log I see that some tests failed and when I start looking at what failed it's like emoji has no attribute replace remember what my update was about I've got it so this is something I would not know if I just pushed my update to Rohite and be happy about it this is an actual problem this is a package maintainer who didn't care about all the rest of the ecosystem but right now I'm looking at it and the real maintainer of the package Python type doesn't know about it they have no idea that this is going to happen how they they are not going to actively go through all the packages their package depend on so when I'm looking at that at this point I'm the most competent person to actually do something about that and I'm going to talk to the maintainers of Python Typer and tell them heads up hey I'm going to bring this update and this is going to break your package you can do it by any means possible maybe opening a bugzilla ticket if there are more packages than one if there are 10, 20 broken packages and you don't know exactly what to do you can help all of them be again buildable you can write to FedoraDevil and say okay I'm bringing this update those are packages that are going to fail this is some guidance I can provide but please in a week I'd like to merge it and maybe you should take a look at it if I'd have 60 packages that fail after checking my impact check that could be a good idea for a FedoraChange so increase the visibility of the issue and if I'm preparing a really big update then most of the time I've read the changelogs and I know or I can find, I'm in a better position to find a migration path so I can help from just telling the maintainers reporting it upstream fixing the issue upstream downstream and providing visibility to providing some guidance I can pick from a big variety of activities in order to help make all of it smoothly seeing that we are going to an end coming to an end I just wanted to tell you about one big copper gocha some packages are quite aggressive when it comes to version and they say that this package only will work with my library which has a version lower than 14 for example my update doesn't bring such version bump but if it did I would expect copper to say ok I have python free reach 15 built in copper this package requires python free reach lower than 14 so we are not meeting the condition so the build fails that is a very reasonable expectation except in copper it would not work it would be false copper has no problem with skipping my just build version with update and going to Koji or Rohide and take the version of the package of the dependency that it needs and in order to mitigate that there is this beautiful shell snippet dependencies all the packages that require my library and check whether they require my library with the lower than character and if they do it will print them out and in my case python reach I will not show it it's a bit slow but actually there are two packages that require python free reach lower than 14 and that would be again my big candidate to reach out to the maintenance of the respective packages and tell them hey your packages will stop being buildable installable and usable when my update reaches Rohide there are some further resources this is a piece of a tribal knowledge that my team has assembled we are trying to write it down and prepare it for further generations so the impact checks are described in our full request review guide also copper has got really usable documentation about the master builds and this presentation with all the code snippets can be found on github this is it thank you very much for your attention so if I have any idea how we could speed up things when it comes to the browsing build logs it is yet to be done the actual build logs the okay so the URL is okay so the URL is divisible I only need my copper name the build number, the package name so I can get all of it from copper monitor I can automate creating this URL where the logs sit then I would need to download all of these logs and somehow assess how much of this output I need to take and maybe provide some automation of results, some aggregation of results whether a certain string is found so that I don't have to just go through 10,000 logs but in some nicer place so yeah it's something that can happen but it didn't yet maybe my colleagues have some ideas some AI engine so yeah maybe this is what will solve all of our problems the issue is that the outputs of the failed logs can be really weird and different sometimes you need to grab more context it's moot to go through the logs of packages that don't build for half a year because probably their failure is totally unrelated to my update it's definitely doable to some extent I don't know to which and none of us had actually much time and cycles to do it so maybe this is waiting for some really brave community folk who would just come and solve this problem for us I'm sorry I hear you really badly so I didn't even grab the core of the question maybe we can talk about it later when I can actually hear you thank you