s Red Hat in vehicle operating system is to rail so there's a parallel though there and that is similar to, we don't have the diagram here but that ASCAS thing, Stream core OS is to CentOS Stream as Red Hat core OS is to rail so it's the different variants of it essentially. Minor detail. And then this is the last one these are, I put them all on one slide because these are three like non-technical SIGs all the other ones I told you about were like doing testing and packaging and building images and these are three SIGs that do non-technical work which is a really great way to get involved if you're not a packager but you like to use CentOS and you wanna get involved you can get involved in one of these. So we have the promo SIG which puts on events Karl mentioned we had CentOS Connect at POSDOM last month or plans like our presence here at scale or managed social media an artwork SIG that does artwork unsurprisingly. It's funny you said for people that aren't packagers because I think that's what everyone's here for today for the packaging workshop. Yeah, well you can become a packager or not. And then Doc SIG which is oddly enough newly formed we had a Docs not a SIG for a very long time and then it finally got turned into a SIG and there's a lot of work happening there. One of the cool things is actually all of the rail documentation is being developed in the open upstream and we're able to use it for CentOS we just have to do a little bit of work adapting it so all of the kind of like the user guides and stuff for administrators we can adapt to CentOS but we have to kind of do the work in the community to do the adapting of that. And we're also working on improving the contributors guide which is not great to be honest and there's various other things on here and we have recently had archived the wiki because it had a lot of issues but there's still some good content on there we wanna pull out. So those three SIGs it's a good place to get involved if you want. Then are there any questions about that or anything on here or anything sent to us? Can you maybe pull back the curtain a bit on kind of how I guess the people aspect of it like are there people that work on Fedora and Red Hat Enterprise for example or are they once they take their curated packages they don't necessarily communicate back to Fedora team or how does that work? So it's on one of the slides if you go back. Yeah. The answer is it's complicated. I'm trying to figure out how to say it in a sentence. It's a little complicated but let me find the one I'm thinking of. No. Here we go. So I mentioned at various points talking about this that the with the changes to CentOS instead of just a handful of people rebuilding rail and calling it as is and can't fix or change anything. Now we've blown the project up with literally thousands of more maintainers because every rail maintainer is now a CentOS maintainer. So there's 100% overlap there. Rail maintainers own their package in CentOS. Not every time, not 100% but there's a lot of overlap with Fedora as well. So you have a package maintainer who is maintaining it in Fedora and let me go back to the... They're maintaining it across the board from Fedora into CentOS and rail. Not all the time, sometimes it's different maintainers but there's a lot of it the case where they have the package, they understand it and they're like, all right, this change in the software is only appropriate for Fedora or this change is appropriate for Fedora and then we're gonna put it in the next rail major which means the next CentOS major or even talking to discussions about putting the change into older rail minor releases and how they weigh that with different red tape internally and other stuff like that but they have a good overview of the whole ecosystem in relationship on how that works. And not only that, a lot of these people got their jobs at Red Hat because they're involved in the upstream software project. So we're not just talking about packages that know how the stuff works in Fedora, CentOS and rail. They also know interact with the upstream project. Sometimes they're in leadership positions in the upstream projects. Sometimes they started the upstream project. So they understand the software there as subject matter expert as you can get and understand the whole life cycle of it and even understand like we can't make this change in the upstream project because of the downstream effects it's gonna have in all these other places I manage it. It's not 100% with that. There are some things where there's disconnect and those are always areas that when we find it we're like, hey, why are you not involved in your upstream yet and try to work to correct that. At the very least get them involved in the Fedora packaging of it. But you know, Fedora does make different technical decisions than CentOS. So CentOS is, people will sometimes say like it, Fedora doesn't have an LTS release, right? Sometimes people will refer to CentOS as the Fedora LTS. And I guess it's about as close as Fedora has an LTS but it does have different technical decisions. It's not, we use, we'll use Fedora 40 as a branch point for CentOS Stream 10, but it's not a copy of Fedora 40. Some good, the most popular example people will talk about is that Fedora defaults to ButterFS as a file system. And that's a decision that, there were Red Hat people that were not fans of that but Fedora's engineering steering committee voted on that and said, yes, this is the way we're gonna go for this. So Fedora makes independent decisions and Red Hat can make separate decisions. Decisions though made in CentOS though, directly reflects what's going on in the next minor version of RHEL. So those will be things that Red Hat approves of. You're not gonna see ButterFS in CentOS until Red Hat's like, yes, ButterFS is going in RHEL. There's less disconnect there but a lot more freedom for Fedora to do things. But you can get ButterFS in CentOS thanks to our special interest groups. Two of them actually. The Kmod SIG has a ButterFS kernel module, the package you can install. And there's also the Hyperscale kernel from the Hyperscale SIG that has ButterFS enabled. So multiple options to do that. More questions. Don't feel like you're holding us up between everyone in cookies. I'm kidding. For the alternative images SIG, do you guys produce other image types besides ISOs like container images or QCAU2 or, and I guess like how do you build all those different images if you do support them? So we could. No one's asked. So if you're interested in images that aren't live ISOs, come and talk to us in the alternative images SIG and we can figure that out. At the core of it, the image build tool that we use currently for our images in the alternative images SIG is a tool called Kiwi. And Kiwi is a very versatile tool that was originally made by the folks in the OpenSUSA project to produce images of all kinds, whether it's containers or virtual machines or cloud images or disk images that auto expand when you flash them onto real computers or live ISOs. Right now we're using them to create live ISOs but if people show up and are interested in helping to make more stuff, by all means we could. Fedora as related to this based on the stuff that's been going on in, I know someone asked about the relationship between Fedora and CentOS in this regard. Well, hyperscale SIG and alternative images SIG doing the work that they did to enable creating what I will call Red Hat Family Distribution Images. That work fed into being able to do the change in Fedora 40, which is that they now use Kiwi to build their cloud and container images. So there's a symbiotic relationship, a very positive cycle of contributions that benefit both groups in a very obvious way. Any other questions here in the back while I'm here with the mic? Not a long walk. Just came back here to rally it up and get questions. All right, then in that case, we will do a packaging workshop but first we'll take a break for about 10 minutes or so. We can chat, socialize and we have cookies and lemonade in the back. So enjoy the cookies. Don't break for too long, people will disappear. Yeah. Not do the workshop. But yes, go enjoy the cookies and juice and stick around to learn how to make RPM packages. All right, y'all ready to make some RPMs? All right, I'm excited. We didn't lose that many people, so. A few people came in and out, it's fine. It's not gonna interest everyone. I know some people mainly interested in the CentOS project overview, that's fine. Some people like packaging and scratches an itch in their brain, being able to deliver software in the native system format. I'm one of those people. So I like teaching this workshop. Yeah, Michelle and Neil, they're the same way. It's not done appeal to everyone but I'm glad that everyone is here wanting to try it out and see if it's something that does interest them. This is gonna be our RPM packaging workshop. Introduce myself at the start but just again for anyone that came in later. My name is Carl George. I actually work directly on Apple that we talked about for Red Hat. Red Hat pays me to work on Apple, not individual packages. The individual packages in Apple are all kind of unsupported from Red Hat but more just making sure that Apple is healthy overall. That's my contact info. I'll put it up on the last slide too if you want it. This is gonna be a hands-on lab. So I hope everyone brought laptops. Go ahead and open up that laptop and type in bit.ly slash hello RPM and that's gonna take you to a hands-on lab that we will run through for this. So the way this workshop is gonna work is that we'll have some lecture and then some hands-on and then some more lecture and then some more hands-on. I've tried to split up the lecture so I'm not talking just at you entirely. Wake you up with some hands-on parts but some of the lecture does go a little bit longer and the last workshops are all workshops where I'll just, me and Neil's gonna help me. We're gonna float around if you have a question whether it's about the packaging stuff and the technical stuff in the lab or just the lab platform if it's not working correctly for you. Just raise a hand, we'll come over and help you out. I don't have a whiteboard anywhere to write down that URL for latecomers, do I? It's fine. If anyone comes in late, I'll just go back to this slide. So everyone present so far has typed in this short URL, right? Everyone that's going to. If you just wanna watch and listen, that's fine too. But the hands-on part is the best part of this lab. But first you gotta listen to me drone on about, geek out about stuff that I like. So what is RPM? It is originally RPM stood for Red Hat Package Manager. They have since changed that to just not be an acronym and be RPM itself. Somebody told me that Scale did the same thing. Originally Scale was Southern California Linux Expo, but to be more inclusive to non-Linux things, they're trying to unacronym that and just call it the Scale Conference. So fun fact for the day. So it's a recursive acronym action. Neil, correct me. It's not just RPM, it's RPM package manager like GNU's not UNIX. So recursive acronyms, recursive acronyms bug me, but anyways. So, but that's just the name. It is the package format for Fedora Linux, CentOS Stream that we've been talking about all day or in the session and Red Hat Enterprise Linux and many others. It's consumed by package managers on your system like DNF. Whenever I first started using Linux, one of the things that interested me the most was this whole idea of packages that rather than just going to download ISI or EXE files off the internet and install them on Windows, having a central repository where you can just say, I know the name of the software I need, such and such, install such and such, that was just magic to me. I love that hook me in, kept me using Linux and eventually I was able to build it into my job doing package management things on Linux. So I mentioned it's scratching an itch for me. I like that part of it. I'm assuming everyone here has at least used Linux before. So I think in the session we described it as, basic command line familiarity is the prerequisite. So you've probably installed a package from the command line before. Why would you want to package with RPM? I know we had a question about other ways of like Apple using RPMs versus other type of packaging technologies. You have all kinds of talk about things like flat pack and snap and other different methods for getting software in a system. RPM is a lot older, but the nice way to describe that is it's a lot more mature. It lets you easily install, remove or upgrade software like any package manager would. It's got commands where you can query your install packages and also verify the files from that packages are in the right place, have the right check sum, right size, all of those things. Each package is not just an archive of files. It is that, but it also has metadata inside in with that archive to describe the package properties. And more importantly, the relationships with other packages, each one of those packages doesn't exist in a silo by itself. It has its dependencies. It has things that it might conflict with. All those relationships, we'll get into those on a future slide. RPM also lets you digitally sign packages to verify authenticity and you can distribute those packages in the DNF repositories like I talked about, formerly known as YUM repositories, still technically defined in etsyumrepos.d, but that's... You call them YUM repositories. No, I'm gonna call them YUM repos because they are. But anyways, one of my favorite parts about RPM versus other system level package managers is that there's a concept of pristine sources. Nothing, I don't have anything against Debian. I think they're a fine project. They do a lot of really good things, but one thing that bugs me about the Debian package format is that they have their system basically intersperses the packaging source with the upstream source in a Debian folder and they manage them together and when a new upstream version of the software comes out, they have to rebase their changes in the Debian folder onto the new upstream software. And some people like that approach. I think it's bonkers and it doesn't work in my brain at all. I like the idea of separation of you've got the upstream source and the package source. And in the RPM docs, or at least in the old, there's a really old book online called Maximum RPM that it describes them as pristine sources and that's a pretty good term for it. It makes it where when a new version of the upstream comes out, you don't have to do get repository management to rebase commits and packages and downstream changes. You just bring in the new upstream tar ball and then try to build it. And if it works correctly, then you're good. It might not build correctly and you might have to make changes to adjust for the new version. But most of the time you won't, you can just bring in the new upstream tar ball as is without picking through the changes in it and build it with your packaging source and most of the time get something out the other end that's usable right away. If anyone, I'm gonna go semi-quick through this. If anyone has a question, go ahead and shoot up your hand and ask. And I'll come bring out one of the mics or see if Sean will do it for me. In fact, let me hand you that. Alrighty, so what actually is the RPM package itself? I mentioned before that it is an archive. It's got the files that are gonna go on your system when you install it, but also metadata about those files. There are two main types of RPMs. You have the slightly misleadingly named binary RPMs that contain the files that are gonna be installed on your target system. The naming around there is kind of the idea that you'd have like a compiled binary, but it's not exclusively compiled binaries. You can have binary RPMs that are what we call noarch where they are architecture independent. They're not a compiled binary. Maybe it's just Fedora wallpapers or a bash script. And it can be, because it's architecture independent, it's designated as noarch and can be installed on any architecture, ARM, X86, 64, whatever. But the correct terminology in the documentation is binary RPMs, even if it doesn't have any binaries. Weird artifact of time, don't worry too much about it. The other type of version, type of RPM is a source RPM. Instead of this files that are gonna be installed on your system, a source RPM has the software source code, that upstream tar ball, like a tar GZ file, and also instructions for building that source code into a binary RPM. It could be as simple as putting that bash script in user bin, whatever, or it might be having to run compiler commands or all kinds of other more complicated steps. So I mentioned the recipe for building the package, or the instructions for building the package. That's gonna be what we call the RPM spec file. It has a preamble section. It defines all of that metadata, and we're gonna go into each of those one by one. And it also has what's called the body. It'll have various sections that'll define stages of the build process. This is also where we have the change log and other things like that. That's also part of the metadata. In that spec file, you can have conditions that give you the flexibility to use the same spec file for say Fedora and Apple, like we talked about, it's just another branch of Fedora or multiple versions of Fedora or multiple versions of Apple or even different operating systems. You could have one spec file that works and builds a correct package on both. These conditions are in there. We'll go into the format of those next or soon. Oh, you can also say, that flexibility lets you do things like only apply this patch when building for rail eight, not rail nine and up, or only on AR-64, skip the test suite, run the test suite on every other architecture because it's broken on that one. Gives you flexibility for tasks like that. So inside the spec file, there is a thing we call RPM macros. That's gonna be a variable that lets you do text substitution. The syntax, it's somewhat similar, if you know bash variables, but instead of a dollar sign, a percent sign right up there. And it has both with and without those curly brackets and it actually is fairly similar to how that works in bash itself if you're familiar with it. You can use the macro without the curly brackets, but if you have anything else on that line that might get confused with the macro name, you need to delineate it and say, the macro name stops here with these curly brackets and then there's some other text. That's the real purpose for that. Both styles can accept parameters that'll influence the output because not all of them are just a straight, this macro has this exact string. Sometimes it can be Lua language is built into the RPM package manager, so it is embedded. So you can actually do Lua things inside a macro that says, with this input, do these outputs and these other things. You can define those. I talked about having them in the spec file, but you can also define those on your system itself. The main locations, it's similar to some other software like, I think SystemD made this a similar structure where you've got something under userlib that is like the system vendor level macros, and then you can have something under Etsy, which is like site local changes for macros that you wanna make, and then you also have a user level one in .rpmacros. You can set up that macro if you put a question mark there at the front of the name inside curly brackets, that'll make it conditional to only expand when it's defined. So if you have the disk macro defined, it'll put in that text there, and if you don't have it defined, it'll not do any string at all. Without the question mark, you would just get the literal characters, percent, curly bracket, DIST, curly bracket. No, it puts the name of the thing exactly how you have it. Maybe that's a very new RPM thing. No, usually you get errors because like the percents. Right, I know what you're talking about. Yes, so like if you have an unexpanded macro inside your build section, it's feeding that into a bash interpreter and says that, what, no foreground control. I think percent symbol and bash does some stuff with foreground, background commands. So you get that error, but what it's literally doing in the spec file is just putting in that exact text of an unexpanded macro, and you don't want that. So if you want it to be optional, you use the question mark. Another conditional form is not just inserting the value of the macro, but to insert some other text. So you could say like, if the rail macro is defined, insert the text dash dash disable this feature. Like think like a compiler flag. You're passing that into the compiler. And so you can say only insert this text if this macro is defined. Ignoring what the value of the macro actually is. Like in this case, rail could be defined as nine, but if rail is defined at all, we're gonna put in this text in that spot. You can also explore those outside of the build process. We'll do, I think that's in one of the labs where you run RPM dash dash eval or dash capital E, and then the exact macro name. And that'll just evaluate that one macro by itself. You can also pass, for macros that take arguments, you can pass them along to it. That's why I have the, you don't actually need the quotes there for like a simple one like this, but if you're passing other arguments, you do need to put them inside the quotes so that bash doesn't interpret them separately. RPM dash dash show RC. That one is going to print out all of the defined macros that your system knows about. Not any in a spec file itself, but all the ones that I talked about in those locations right there, it'll print out all of those together. So you could actually see that, oh, I've got this thing defined in my home directory and this one's defined system level and so on and so forth. So some of the really common macros that we'll have is for file system paths. So like underscore bender is going to evaluate out to user bin, datadir goes out to user share, sysconfdir expands to slash ETC, which I always hate that one because what it evaluates to is shorter than the number of characters of the macro itself, but that's my OCD acting up, so. Yes, but still Etsy with the macro around it is still longer than just putting slash Etsy. But anyways, I digress. You can also do operating system properties. Like the rail maker I talked about, that's going to be defined to the major version. Fun fact, since this is a CentOS classroom thing, rail is defined on CentOS also. So on CentOS Stream 9, the rail maker was also defined as nine because they're really dang close to each other and I keep telling people of that, not everyone believes me. Yeah, but either way, nine, so you can have, if you use this conditional, that is saying that whenever it's major version rail nine, do this thing or insert this text. So the disk maker that I've been mentioning in a few places, that is a special one. If you haven't talked about version in a release yet, that's coming up, but this lets you insert the string .el9 and that makes sense later, but the short version is that when you're building, using the same spec file to build for say, rail eight and rail nine, the resulting package in the release field is gonna end with .el9 or .el8 to indicate which thing it was built against. El9 is literally, it's a little misleading, it evaluates to one, but think of that in the like, bash one true type sense, like if one, do this thing. More common macros are build process helpers. If you've been doing software on Linux for a while, you might have found a project that wasn't in your repos and you wanted to do a custom compilation of it. You might be familiar with the traditional dance of .slash configure, make and make install. Real classic way to design around and make file. Those will usually take extra arguments to help shape where it is. Like if you just run those without any arguments, you're gonna usually end up with stuff in user local, but that's not what we want for an RPM package. So we'll have, we have these macros like configure and make build that actually pass in all of the predefined arguments, distribution wide to install things to the right locations like user bin instead of user local bin and GCC flags and other things like that. It's better to use those macros because you wanna use the standardized distribution options for all of that compilation and installation and not just whatever you come up with on the fly that may not account for everything that's necessary. There's also the auto setup that's, that is gonna be related to before you get started doing the build process and we'll get into the sections of that process later. The first thing you have to do is extract that tar GZ file from the upstream source code and maybe optionally apply patches that you need onto that source code. That's what the auto setup maker is gonna do for you. Some more ones, what I affectionately refer to as the legacy Python helpers, they still work, they're still good, they're not that bad, but PY3 build and PY3 install, that'll correspond to Python 3 setup PY build with some extra flags and install the same way. If you've done by hand installation of Python software, you might recognize that pattern. We have more modern ones now. They're in Fedora and also in CentOS 9 and Rail 9. They're related to the, if you're familiar with Python projects moving from setup PY files to pyproject.toml files, it's tied in with that. The pyproject macros can also find and interpret the setup PY stuff. So in Fedora it's better to always use those pyproject things. They work great in Rail 9. They don't have all of the features in Rail 8. There's some ongoing work to get those all back ported and working, but there's also a thing in there called dynamic build requires that is not gonna fully work on eight. Frustrating Neil, very much. Other patterns you might be familiar with from other software. We have macros for CMake, for Mason. These are called, I think these are referred to as build systems, which is not the same thing as like a package build system like Koji and Fedora. Overloading of terms. It's frustrating and difficult. But yes, they're, build framework. I just decreed that's the new term. But no, they describe themselves as build systems, I think. So yeah. There's also test suite helpers. There's gonna be part of the spec file where you can actually run the upstream test suite. They're unit tests. Hmm? Which one? These are just a few examples. No hecklers. Go. That's my job. So you've got some common ones. The PyTest maker will run the PyTest command with packaging specific options, similar with CTest and Mason test. So good. I get to take a break from talking at y'all and let y'all actually get to do some hands-on stuff and wake up. Right after this is the start of the lab. But what you're gonna do in there, there's a tool, a command called RPM Dev Setup Tree. It's in a package called RPM Dev Tools. And that is going to set up several directories that the RPM tool chain expects to exist. In your home directory, you'll have RPM build and then build RPM sources specs and source RPMs. And during the build, during the future labs, you'll see how each of those gets used. And that is the wonderful hands-on lab we're about to start. So that URL, did anyone need to see that URL again? Did it come in later? No? All right. If you're just shy, just ask me and I'll come by and tell you that you are all butt. Go ahead and load that up, click the start button and follow the on-screen instructions. Unless you're an overachiever and already clicked next and started without me. Some people, there's always someone that does it. Like, I'm starting the first lab and they're like, all right, I finished all four labs and I'm like, you're supposed to wait for the, the lecture goes with it and teaches you as you go. But anyways, if you have a question, whether it's the lab platform itself or the instructions in the lab, this one's really straightforward. I think it's just two commands. But after that, if you have any questions in any of the lab, just raise your hand. I'll come around.  We're not shaming you in front of the class.  And I need to make note of the time because this one, we'll give this lab about five, this lab's quick, maybe about five minutes in about five minutes. If anyone's still working on it, just raise your hand and I'll wait a few more minutes. But this one should be pretty quick for you. Just an FYI, Neil here is going to be my sort of co-presenter helper. He's also gonna float around and be available for any questions people have. Anything not working correctly. So feel free to ask him or me questions. I think a few more people are working. So it's been six minutes, but we'll give it a couple more minutes just to make sure we don't leave anyone behind. Another FYI, this lab actually is staying online, I don't wanna say permanently, nothing's permanent, but it's not getting turned off at the end of this workshop. So if you wanted to take this and do it again or share it with a friend, you can. I think the lecture that goes with it is really valuable, but I'm trying to make it where the lab itself has enough instructions for someone to just go in and do the lab with those instructions there. So like, for example, like a friend of mine didn't bring his machine. So if he's gonna later on go to that same URL and do the lab, but just listen to the lecture right now. So that's also an option in case anyone didn't bring a laptop. I was hoping we had notes in the program about bringing your own machine. We didn't get any hardware provided, but I don't see it listed there. So either messed up or didn't print it in the right field or whatever. Quick show of hands, no shame if you're still working on it, but I just wanna know if I should go ahead or wait a few more minutes. Is it, is anyone still working on this lab? Got one, two, okay, three. All right, yeah. We'll wait a few more, let people get all to the same page and then before we move on. Just do the RPM Dev setup tree stuff, hit the next, don't go on to the next part because we have lecture that goes on. Stop being an overachiever, Sean. Only click next once. Oh, so you're caught up, cool. Okay. Like the one time I did this lab at Summit and during the first lab, somebody just had to tell me like, all right, I finished all like four or five labs and I'm like, cool guy, good for you. Your first place, move on. Awesome, I think everyone that raised their hand said they're good now. Keep getting blinded by the projector. So, let's do a little more lecture. I gotta bore you with my voice a little longer. So the lab we just did, that gives you the basic tooling that we're gonna use. But now let's talk about the spec file format itself. So the first section is called the preamble. This is gonna do with the build process at all. This is all about the metadata for the package and that includes like package relationships to other packages and we'll get into those. But for just the plain metadata fields, the name one, somewhat obvious, it is the name of the package. Usually you're gonna pick that name based on what the software is called upstream. It doesn't have to match the upstream. So you could do it differently. Simple example, the upstream might always capitalize their project name but in the package you wanna make it lowercase. Yeah, there's lots of ways to go about it. I think the Fedora packaging guidelines have a preference to lowercase everything unless the upstream project specifically asked to keep a certain capitalization, which is fine. But there's lots of flexibility there but yeah, it probably will match something upstream somewhat just so people know what the package is but what it does have to match is the file name of the spec file. So if you're packaging foobar, your spec file needs to be foobar.spec. The version of the software, that is going to be, I think I touched on this in one of the Q and A questions earlier but the version that's gonna correspond to the software's version upstream, what they released the version as. The release is basically another version but that is the version of the packaging you're doing and you use that separate version to distinguish between different builds of the same upstream software version. And I'll show that in some of these examples but these three fields together, name, version and release, they form around like the Fedora project and other places that deal with a lot of RPMs, you'll hear the term NVR thrown out a lot and that literally is the name, version and release jam packed together and that forms a really useful identifier like if you're explaining in a bug report which what version of the package you have installed typically you'll use that NVR full identifier not just the version, the upstream version by itself. Here's some examples. GNU AUK package is named GAUK and then we've got a hyphen, the upstream version 421 and then another hyphen, a 4.EL8, that.EL8, that is the disk maker we talked about earlier expanded to that so when you built, you could build the same spec file and you'd get all of that .EL8 and then all of that .EL9, two separate packages built for different like GCC lib and other things like that, not GCC lib. Glib 2 is what I was thinking of, different libraries, they'll build for the different, that indicates the target you've built for. You can't just do the split on the hyphen though because package name, it's also valid to have a hyphen in the package name. I'm getting ahead of myself here but you can have a package name vert what with a hyphen in the middle and then your version and your release field. The way I'll do it if I'm doing something custom Python for like parsing this is Python has an R split, split from the right and then two fields, so from the right to the first hyphen is the release and then from the right to the next hyphen is the version, everything else is the name and the one in the middle that's just showing, what's that? Yeah, pythonsstring.rsplit, Rsplit 2. Anyways, and then the middle example, I just had that there to show some different versioning.  There's a terminology called semantic versioning that that looks like and I would not ever trust that an upstream is doing it just because it looks like it. Semantic versioning is a really good system that a lot of project use something that looks like that superficially that doesn't work like semantic versioning rules at all. You might native, yeah, you might understand, you might look at that and say like, oh, I understand that there's the major version and then like the feature version and then like the bug fix version and that is what some projects do. That's close to like semantic versioning. It's a whole schema, I guess, or like standard and it's great. I love it when projects actually follow it. A lot of them don't. For example, Python, love me some Python, but Python, the first two digits are basically the major version, like 3.12 or 3.13, right? First two octets, they're not digits because that's something to point out. Version numbers aren't really numbers, they're strings. I had this conversation at a previous employer with someone that really bugged about the fact that like something.10 evaluated differently than something.2 and I'm like, well, because it's a string of one zero not a decimal of dot one zero. And so, yeah. And because upstreams do all kinds of different versioning like you might have something like the TZ data project that releases 2024A and that's not alpha, that's just A and then B and then C and then D on and on. Yeah. You might also have alpha and beta identifiers inside the version stream. There's all kinds of ways upstreams can do this. RPM has pretty good logic on around evaluating that. It has to understand that so it understands what RPM's an upgrade for which other RPM. But there are still things that can throw it's sorting out of order and we'll talk about that with this wonderful thing called epoch or epoch. This is a third version field and what it's used for. So, huh, the fourth? The name's not evaluated for the version comparisons. This is a third one. So you've got the version and release that I talked about like the upstream software version and the package version sort of the release that you're doing of it. But let's say you have an upstream project that decides that we were using versioning of 2024.01. But we don't like that anymore and we want to switch to proper semantic versioning. Hooray, everyone loves this. And now we're going to be versioning 1.0.0 for our next release. But RPM's evaluation is going to look at those octet by octet or section by section and say that 2024 is bigger than one. Therefore, this one on the left is a higher version and 1.0.0 is not going to show as an upgrade when you do like a DNF update or DNF upgrade on your system. Epoch is a bigger hammer. It is a super version in the spec file where you can say I don't care what the default sorting is, this thing with an epoch is bigger than this other thing without an epoch or with a higher epoch. No epoch is basically an implied zero epoch. Normally you wouldn't have to, you don't have to put epoch zero in your spec files. You normally shouldn't have to touch this. It is few and far between where this is actually needed. Some people make mistakes and I've seen it before. Some packages you'll see in Fedora and CentOS and RHEL have a really high epoch, like 70. And there's no way that an upstream project has changed their versioning in a non-sorting way 70 times. It's just not. But there were, I forget what, I think Pearl has some like 40 or 50 epoch. But there's historical times where someone mistook the role of epoch in release and they were bumping the epoch every time they made a packaging change, which technically works, but it's like a misuse of resources. Yeah, stuff like that. So usually it's a mistake when it gets to a really crazy high versioning. More likely you'll have just a no epoch at all or an epoch of one because one time the project made a non-default sorting version path. It is also possible that you might need to use, I had to use an epoch the other day. I switched the package, Python doc op, it's basically dead upstream, hasn't been updated in like six or seven years. No responses from the maintainer, it happens. Sometimes people, let's say the lottery factor. People talk about the bus factor, which does actually happen, sadly. My manager tells me use the lottery factor because that's a lot more positive. Yes, but they haven't touched the project in a while. That's fine, somebody made a fork and somebody asked, hey, for this package, can you switch it to the maintained fork? It's a lot of people involved, standardizing around it. It's great. And I was like, just makes perfect sense. I've done this in packages before, switching to a maintained fork, but I didn't realize that with that switch, they didn't keep the Python metadata name the same. They actually switched it from doc op to doc op ng, corresponding to what's on PyPy, the Python package index. And so that change caused some unintended side effects with the upgrade. And so I was able to go from, I think it was from version six. I upgraded it to version nine in the fork. And then I realized this is actually broken and not compatible software. I need to undo it. So it wasn't an upstream mistake or change. It was my mistake. And I was like, I need to actually force the upgrade path to go from six to nine, back to six. And I did that with an epoch. So now epoch one version six is higher than version nine without an epoch. That's what it's for.  We won't have to use that in this lab and you almost should never have to, probably not have to use it, but it's good to know that it exists if you need it. Yes. So also in the spec file preamble, we're gonna have these other fields. The summary is exactly what it sounds like, short one line thing. You'll get, during the lab, you install the package called RPM Lint. We'll use that later in future labs, but it'll complain if your summary's longer than I think 80 characters. So you wanna keep that fairly short. A lot of upstream projects will have like a short description and a long description. The summary would correspond to the short description from upstream a lot of times. But this is up to you.  The license, that is gonna be an identifier for the license of the software, like MIT or Apache 2.0, things like that. Small note on the license thing, Fedora is in the middle of a project to standardize, we had our own system for a long time because there was no standard for short identifiers. There is now the Linux foundations back to thing called SPDX. And Fedora is standardizing on using SPDX identifiers across all of our packages. Still work in progress, there's a lot of packages. RPM Lint in CentOS 9 and RHEL 9 doesn't understand those identifiers yet. So in the lab, for the sake of not getting an extra linter error, we're using the old identifier. But if you start getting involved in Fedora packaging, know that those identifiers are technically wrong now and against the guidelines. Coincidence. Neil said that MIT is right on both because the legacy identifier and the SPDX identifier MIT is litter just MIT. Yes. Unless you're using one of the weird MIT variants that used to be called MIT that is not MIT by itself anymore. Anyways, that's a tangent. You've got packages like geeking out on stuff. But anyways, so summary license URL. That's just what it sounds like. It's a URL where you get more info about the software, either a project's homepage website or the GitHub repo or GitLab repo, whatever you wanna call it. Just where you would go to get more info. The homepage of the software. BuildArch, that is, I talked earlier about the binary RPM name being a little misleading. BuildArch, normally you don't have to define it. It's gonna be defining your packages you build based on what architecture you're building the packages, the system you're using to build them. But if you wanna build a NoArch package that is valid on anything like, say, wallpapers or bash scripts that are architecture independent, you can manually set BuildArch, NoArch and define it that way and then you'll get a package that isn't designated as specific to particular architectures.  If you skip defining this, you get a package that only works on one of the other arbitrarily set in the metadata even though the actual contents would work on both. So this is the way you make it. You actually identify to the package itself that you can work on multiple architectures. You don't have any architecture specific files. More preamble sections. The source, this is gonna be a file name or a URL to a file that you need to build the package. You'll, this is where you'll, the first one you'll see usually is the source code archive link like a GitHub download URL or like if the upstream project has their own file server they've published their archives, their tar balls under, you could use that. You could also use it for things like a default configuration file that maybe it's just locally with your spec file. Maybe it's at a URL somewhere online. It can be either one. Then you also have patch tags. These are similar to source. It's either a file name that you have right there alongside your spec file or a URL to a patch file that you're gonna apply to the source code to make small changes. You can use both of these tags multiple times and optionally suffix them with digits. In Fedora, so the suffix with the digit is actually mandatory on rail seven and rail eight, I believe. Rail nine it's not, you can use unversioned. It's a little, I think it's a little cleaner to use the unversioned one and I would only mess with the version source and patch if I had to specifically do something with one of the sources or one of the patches. If you've just got a source tar ball in three patches, you don't have to number that because that can all be applied automatically. If you want one of your patches to only be applied like on ART 64, then you might wanna number them so you could say only apply this specific patch number. Things like that. So now getting into the other, more important type of metadata I think, your package relationships. So the first one is build requires. These are gonna be packages you need in order to build your package. Also known as build time dependencies. As opposed to requires, that's gonna be packages that you need to install this package or runtime requires. Good difference there. And it's important that you, those are separate fields. Some people say like, oh, well anything that is required, I'm gonna have it build time. And that's not true. You can have runtime only dependencies that aren't necessary to build your software in any way. So you have control there and you, it's a little duplication if you have a long list of dependencies but it's not that bad. It's better to have it a little bit more explicit I think. Recommends is a slightly newer one. I believe it works in CentOS 8 and 9 but not seven. Of course you remember the exact version that's introduced in. But yes, recommends it's called, what's called a week dependency. It functions like or requires like a runtime dependency and it'll be installed by default. But the difference is that you could actually remove it and not remove the package that had a week dependency on it, making it an optional dependency. You could also configure your system to skip the optional, skip the recommends and optional dependencies anytime. Supplements is really cool. That is what I call a reverse recommends. There's no requires equivalent to it but basically with the supplements, the example I like to give is think about, think you have a piece of software and then you have plugins for that software and those plugins change over time. You package new ones, add them, you might remove some. If you use recommends for it, the main software to pull in all the plugins, you'd have to enumerate all of the plugins in the main software spec file and then if any of them change, update that list of recommends in the main spec file. Alternatively, you can use supplements and have every spec file supplement the main package and then if you remove or add a new package, you don't have to update the main package at all because it's working in the opposite direction. It's saying I'm an optional dependency for this thing and it just gets pulled in and recognized. Conflicts, that means packages that can't be installed at the same time. It's better to avoid this one if you can but sometimes it is unavoidable kind of like the epoch thing. Usually where this will come in and then to play is file path conflicts. You might have two completely unrelated software programs that both install, they both install user bin foo or user bin convert I think is one that we have to deal with that this software has it but this other software completely unrelated also has it and you neither upstream wants to rename their program but there's a namespace collision so the only thing we can do, if you don't do anything, DNF will let you try to install them. It'll start that transaction and after it's downloaded the RPMs and tries to install them, it'll abort and say I actually can't install these because they both have a file at this spot and that's not gonna work. You can tell DNF ahead of time that these two packages can't be installed at the same time with that conflicts directive so it will actually tell you before it downloads those RPMs. It'll look at the metadata, say nope, these two conflict and that breaks the transaction you can't install them at the same time. Maybe more common example than just an accidental name collision would be Forks of Software. You may have one software and then another somebody else forked the software, they changed the name of the project but they still have the same command and then maybe it was a contentious forking, maybe not and neither projects like nope, we're owning the name or even we have to have this file path because the way the software is written it expects this in every artifact of this other thing. Different real world examples I'm bringing up but in the end you can make those two packages conflict with each other so that way DNF understands that they can't be installed at the same time. Obsolete's, that is a relationship where you're telling the system that one package is going to replace another package. That might be used for if you're renaming a package like let's say, I know this isn't how it worked but let's say you had a WGIT package and you wanted to replace it with the WGIT 2 package. I know Neil's gonna want to jump in because that's not how the WGIT 2 project worked, that's not what it was, but anyways, if you wanted to replace one with the other you could use the Obsolete's in the second to say this is going to replace the first package. Sure, eventually, eventually it will. Okay, so yeah, so Neil brings up the fact that the DNF project itself is undergoing some changes where DNF itself is version four. They've been working on a DNF, a new major version five, but the way they're doing it is basically putting five in the project name and in the binary name and they're gonna have Simlinks, user-bind DNF is gonna be a Simlink to user-bind DNF, DNF five. But it is also at the same time replacing an older project called MicroDNF that was used in containers. It was basically a DNF with no Python dependencies that could only install packages. Not, I don't even think remove them or do other more advanced stuff but super lightweight, great for installing stuff in containers. DNF five is part of the DNF five rewrite is to remove the Python dependency, the rewrite mostly in C++. There's still some Python stuff, but the core executable is going to be slimmed down and replace the functionality of MicroDNF entirely. So they're having DNF five, Obsolete MicroDNF. That's a good real-world example of it. Provides is another relationship. This one's cool. It lets you allow other packages to refer to your package by another name. And that's kind of a vague way to describe it but the real-world example to make it concrete is you could have a package that needs a web server installed but you don't care which web server, you just need a web server. You want a web server to be installed for this software. You could have web servers in Fedora and CentOS actually all provide the name web server. So the HTTBD package, the Apache web server provides web server, just the name web server. EngineX provides web server. Catty provides web server. So you can have another package, require web server and have that dependency satisfied by any of the other web servers. Doesn't matter. If you don't already have it installed, it's, I don't remember how exactly it picks one. I think it might pick the one with the highest version. Okay, yeah. It lexigraphically does alphabetically on the name first. At one point I thought it picked the one with the highest version but that's not correct. Neither does picking one by first alphabetical either, but. A, A, A, A web server. Yes, but it allows you to install the web server first or in the same transaction explicitly name which web server you want to install and then that package will provide the web server name that satisfies the dependency of the other thing. So that's where provides come in. It also comes, I'm not hitting everyone. I've never, legitimately never heard of that. So I'm up here learning things as I'm giving it. Apparently there's one called build conflicts and I'm gonna have to look into that later because it's not a RPM 101 level thing. The thing you gave as an example for requires for conflicts at runtime where you have like multiple providers the same thing but you know that one is not good can also apply at build time. Because like for example, a good example of this is the user bin convert as you started and stopped with yeah is owned by image magic, graphics magic and like four other different things but like they change only in our package upstream they still do user bin convert. So yeah, no, you get to have fun with that but if you wound up having a situation where there's multiple theoretically identical providers that are actually not, you need a way to tell it not to pick that and so build conflicts is the way you do that. So the same way you tell it to not pick something at runtime you can also tell it to not do something at build time. I would just do an exact build requires. Yes, more advanced things. And this is not gonna be conclusive all of the possible tags. If you are interested in those rpm.org has the full docs for every possible thing. This is entry level 101. So no more obscure new things for me. Don't make the presenter look bad. Sorry, this might be an obscure one as well but in Santos they have to care about multi-lib conflicts as well like two packages like 64 bit and a little bit and you try to update one without the other and then you get weird conflicts. So I don't know a lot about multi-lib. I'm sure I could hand the mic to Neil and he could talk for 30 minutes about it but we're not gonna do that. Yes, multi-lib is a thing where specifically for library so that way you can have both like a 32 bit and 64 bit version of the library installed at the same time. I think for the most part those get the conflicts worked out automatically because there's user lib versus user lib 64. So I think most of those are just resolved by being in a different parent directory. Yes, data files in common could cause a problem there and I don't know how that's resolved in multi-lib actually. Let's talk about that more afterwards. That's good question beyond the scope of this workshop. Yes, let's just get rid of 32 bit. That solves the problem. Alrighty, so more fields in the preamble. We've got the description. I mentioned before that some projects will have like a short and long description upstream. This is where you'd put the long description. It can span multiple lines. It can be multiple paragraphs if you want. They don't really need to be that long. Really the target for this is when someone does, if you do just like DNF search on the command line, you'll get that short summary listed next to the package name. If you do DNF info on the package name, you'll get the summary and the longer description. Some software doesn't really describe their software that much and so you just use the summary with the period at the end for the description. Or make up your own. A lot of times I'll try to get like the first paragraph of the upstream read me or maybe massage it down to the first few lines, something like that. But it's whatever you want it to be to describe the package in DNF info output basically. And then a little bit more advanced thing that you can do is what's called sub packages. You might have seen this before if you've seen, I'm trying to think of a simple library. Let's say you've got libpcap and then you also have a libpcap devil package. Those are, it's the same software but you're splitting up the development files into one package and the main library files into the other package. Some distributions keep those all in one package and get bigger packages because of that. But in RPM, it's standard practice to split those things up for the use case because everyone that needs the library present doesn't need all of the development files present. So you would have a devil sub package where you place those different things and you control that by adding a package and description section in that preamble. In each of those, the package one starts a new preamble and then the description just goes along with it. So you could say these are the development files that go along with this and it has this other runtime requirement you have to have installed. The name there, one interesting note, that's not gonna be the full package name but right there in that format, if you just put a name there, that'll be appended to the main package name where you have like foo and foo-devel. You could also do an arbitrary name completely separate from the main package name if you add the dash in flag in the middle there. So you could say like you have package foo with a package bar as a sub package and it's not immediately apparent that they're related because it's not a suffix. So you usually wouldn't wanna do that. You'd have your sub packages named with just the suffix there. That's a little bit more advanced. I don't remember if we have that in the lab or not. Oh, we do I think for the Python lab. No, actually I'll take that back. No, the Python lab doesn't have that. So you won't see that in the lab but it is good to know about. And if you've seen those like development packages or separate packages like that, that's where that comes from. That's how you define that. So the spec file body. This is where we really get in the meat of the build process. The prep section, that's gonna be where you unpack the source code archive, the tar-gz file. You need those files extracted and then you would also apply any patch files to those. Like let's say you have a patch that changes, fixes a bug from upstream or adds a fix for security problem. Anything like that, you're gonna be applying that during the prep section. And we have that helpful auto setup maker I mentioned earlier will apply all of your patches for you without you having to manually go through with the patch command. It's really useful. The next section is build and it's exactly what it sounds like. That's what you're gonna run to build the software. Those helper makers I mentioned earlier like make build and make install, those would correspond to the build and the next one, the install section. Whereas build, that's where you're compiling the software usually. Install is where you're going to place those built artifacts into a directory tree that, what we call the build root, but it's going to be simulating the file system structure on the target system. So inside your build root, there'll be a subdirectory, you know, user bin, user lib, all those things and you'll place those files where they need to go in that subdirectory structure and then that becomes top level structure in the target system when you install it. And then finally, there's a check section. That's not exactly building the software, but that's where you can run any kind of commands to test the software. Upstream unit tests work really well here. You might want to do like a functional test where like you run the command and check for some kind of output. Really anything you want to test the software functionality. Two more, I think we're gonna get to the lab soon. The file section. Files and change log work differently than those other sections. The prep through check section, those were basically bash scripts. These ones, files and change logs, these are text and in this particular format. The files one, that is a list of the files that are gonna be installed on the target system. Like user bin foo and then user shared doc foo, whatever files you're gonna have on the target system, you just list those, the full path to those right there. The change log, that's in a specific change log format. You'll see it in the lab. That's where you're gonna record changes that have happened to the package between the different versions and the different releases. You don't have to be overly verbose there. It could be as simple as updated to the new version. I've seen some packages where they try to jam in the entire upstream change log in there and that's not what that's for. Fedora package guidelines actively discouraged against doing that. I might actually forbid it now that I remember it. Yeah, it should be fairly, fairly short. You could link to an upstream change log, things like that, but it's more if someone runs RPM-Q-dash-change log on their system, what do you want them to see? Do you want them to see 1,000 lines of upstream commit messages? Probably not. You want them to see like, hey, there's this new upstream version and I added this sub package or I patched out this problematic sub-commander, plug-in, whatever. Things that would be relevant to the end user immediately. Sure. Sure. Where's my micro-runner? I've got a lab helper and a micro-runner. For the files macro, is there a way to automate a long list of files that need to be installed? Let's say your make outputs 100 files or something like that. Yes. Yes, there are lots of ways to do that. So I know for example, in the file section you could do, we haven't talked about this slide yet, skip ahead, so in the files, at the file, the name, the start of it, you could do percent files dash f and then a file name, a local file name that's there in the build process. If your build process can write out those list of files, I know system D makes really heavy use of this in their package. That's another one, yeah. So system D does that. So during the build process, system D writes out that this component of system D has all of these files in these locations and you can save that output into a text file just during the build process, not part of the package installation. And in the files section, just say files dash f, this temporary file during the build and it injects everything of that file contents into that. There's other macros, another way, instead of listing every file exactly, we'll get into that, yeah. There's a recursive way in there, I'll talk about that in the next slide, but there's another question, but there's several different ways to go about it, yeah. No one more. Yeah, for the back porting, is it recommended for tracking purposes? Because in Git, you can cherry pick commits and it'll tell you this commit was cherry pick from this upstream commit. Do you recommend in the change log to do something similar?  Yeah, it's a little bit of dealer's choice. There's definitely best practices around it, like for example, your back porting a patch that fixes a specific CVE, you would wanna put that CVE identifier. People look for CVE identifiers and change logs. That's a great example of something you would do. I wouldn't just copy the upstream commit message in entirety and put that. Even if it's one line, that's probably two verbose. What I personally, what I'll typically do if I'm pulling a patch from upstream is I will write add patch to change or fix this thing, you know, back port it from upstream or something along those lines or even better rather than just say added patch for this thing and then a link to like a Red Hat Bugzilla number with more information. So it's really good for putting URLs like the bug number. It's really good for that. I've seen some packages where they will reference like an upstream pull request number and say like, this is a back port of this upstream pull request that fixes this thing. Maybe they also have a CVE identifier, maybe not. But yeah, it's dealer's choice. It's whatever you wanna put there. I would say best practice is the shorter the better. You want it to be very terse and just a quick something you can look out on the command line and understand. No, no, no, the patch content itself doesn't go in the change log. The patch itself is in that patch file, the one we referenced. Oh yeah, okay. Yeah, there we go. You would have the patch in a file like if you did get format patch, that gives you a patch file in like manless format. You save that as a patch file and during prep it applies that to the source code and in the change log you just describe what you did in as high level term as possible. Like I'm back porting this patch for this thing. So some real world advice here for what I usually do is if I'm back porting a fix or adding a patch or in a feature or security bug or whatever, I will take the commit back as a patch, put it in as a patch line and I'll put a comment line right above and save from and either put a URL or the commit ish or the reference pull request or whatever because that detail is useful there when you're looking at the details. In the change log I'll just reference as Carl said the very shortest form possible to explain it because the way you got to think about change logs is that it's some dude who's possibly looking from an SSH terminal remotely on possibly his phone and needs to be able to read it and from that perspective you should make it the shortest, tercest while still being understandable that you can possibly do because that dude is not going to suffer through reading a whole bloody git log on his phone. The level of a verbosity is something to think about there. Another follow-up point of what he was saying, referencing like an upstream pull request or commit, in Fedora policy, Fedora and Red Hat in general across the ecosystem of distros we have, we have a policy of upstream first and what that means is that we don't wanna have changes that are just special to us. We work upstream first. I mentioned that a lot of the rail maintainers are also involved all the way through CentOS, Fedora and the upstream projects. They're doing stuff in the upstream, they're getting those changes there and everything else is like soft forks of management of what combination of these upstream commits we're gonna do. It's really what it comes down to. But whenever you're doing a back ported patch, you, in the Fedora packaging guidelines, in line with that upstream first policy, if you're adding a patch in Fedora, there's a should requirement. There's an RFC, I forget the number, but should versus must terminology where you should, unless you have a really good reason, have a reference to where that back ported patch came from. And if it's something you came up with yourself or got from somewhere random, you need to work to get it in the upstream project because it's very annoying to have a huge long list of patches. Every patch you're maintaining in your package is making your soft fork, a little bit more of a soft fork, a further divergence and that can be very difficult to manage. The much more responsible way to handle it is to get that upstream, I maybe even get it from upstream in the first place. So that way the next version of the software that comes out has that patch built in and you just drop the patch file. That lets you stay at zero or one or like single digit patches and drop them as time goes on. But then for some longer life distros like CentOS itself, you're gonna have the opposite of that where I do have to keep system D on the same version or the kernel on the same version and you end up with hundreds or even thousands of patches. But those are also, those are going upstream first. They're not just secret sauce, red hat things or CentOS things in that case. They are things that are from the upstream project. There's not value in keeping those two CentOS or RHEL specifically or even Fedora. Fedora has the same, Fedora can back port stuff sometimes. You might see it where new version of the software, it has the fix that we want security or otherwise, but because of the way that change stuff, maybe config file change or a library change, it's only appropriate to put that into Fedora raw hide because Fedora releases also have a updates policy where you're not supposed to make big changes. It's just shorter life cycle. The 13 months versus like five or 10 years. So older Fedora releases might get a back ported security fix that get dropped in the next Fedora release because they're included in the next version. All right, so that answer the question mostly. We'll keep going. Where were we? The file section, I believe. Yeah, so different adders, adders last point, but yes. In files, each of those lines, you can proceed it by a different attribute. The durr, percent dir, that'll just say own this directory that I'm giving you, but not its contents, non recursively. If you just listed a directory, by default, it's going to include the directory and everything underneath that directory. Think like user include foobar. If you just put that in there, it's gonna include that directory and every file underneath there, like the PC file and other stuff in there. But if you did the durr on user include foobar, you won't get any of those files underneath the directory just the directory itself. And you may think, well, there's files in there that I did in the build process. I want those pulled in the package somehow. Where that really comes in handy is where the package splitting thing, where you want some files in this sub package and other files in this other package, durr gives you a lot of flexibility so you can say the top level package owns all of the directories and these sub packages own the files in those directories and they may be different sub packages owning different files there. So it gives you a lot more granular control. Config lets you mark something as a configuration file. And it has a really weird kind of dumb default and I don't know the historical reason why, but if you just put config in front of the file, it marks it as a config file, but whenever you update the software, if the config file changed in the output of like the build system, it's gonna replace all the local configuration when you install the package, which is almost never what you want unless the config file format has completely changed and the local config file is basically worthless. When it does that, it will save the old config file that you're placing as a .rpm save file. So it's not completely obliterated, but it's moved out of the way and might surprise people. Almost always you'll want to use the next one, config with the parameter of no replace. That means, and you've probably, if you've ever seen RPM new files on RPM based systems you're managing, that's where that comes from. There's a new config file thing. Could be as simple as they changed a comment at the top or they added new default directives, new sections, things like that. But that will create the config file in with a .rpm new suffix, which is a signal to like the system maintainer that, hey, you've got a new config file and it's up to you to merge in the changes, but your existing one is probably still compatible with the software and things will keep working. But if I forget the exact, I think RPM comp, there's a tool that lets you find all of your RPM new files and RPM save files on your system and go through and merge them and manage them or you just do it by hand. But yeah, config no replace is almost always what you're gonna wanna use. Nobody likes their config file getting clobbered on purpose. Adder is another thing you can prefix the line with. That lets you set the non-default permissions or ownership of things. All right, some attributes will take, will accept relative pass, which can be, which will copy the file into the build route itself. Most of them you're expecting to already be in the build route from your install section and you're just listing where the file is. But license and doc, you can give it a relative path to like the license file in the tar ball and that macro will copy it into user share licenses or user share doc and also mark it appropriately as a license file or a documentation file. For a long time, we only had the doc macro and the difference between these is that you can skip documentation files. You can't skip license files because most licenses have a clause that you have to distribute the license with the software. So that's what the difference, the key difference is there. So creating spec files. You may want to create these from scratch. You could also copy a similar spec file and adjust as needed. That's a real common. Some text editors have automatic templates like for example, if you use VIM and you just type VIM foobar.spec it'll inject a basic skeleton of an RPM spec file just automatically it understands the template. Other text editors have similar mechanisms. The tool you're gonna use, actually you don't use this one in the lab, we use another one, but there's a tool called RPM Dev Newspec from that RPM Dev Tools package from the first lab. That'll create a spec file from a template also and just output it. Similar story with change log entries. I mentioned that's the description between each version and release of the package as it changes over time. You could write them by hand. You could copy the previous change log entry and then adjust it. You could use text editor plugins to inject it. But then there's a tool you will use this one in the lab called RPM Dev Bumpspec. What that's gonna do is it'll either add the change log entry with whatever comment you pass that command. And it'll also at the same time adjust the release and maybe even the version if you need to along with what changes you're passing into that command. And you'll see that in, I thought the lab was the next slide. It's gonna transition right into. So, but there's also about building RPMs. We're gonna build them with the RPM build command. That expects the directory structure we made in the first lab, the RPM Dev setup tree. We've got various build modes. BS is going to build the source RPM from spec file and sources. BB is gonna build your binary RPM from spec file and sources. BA is gonna build both. And then there's also another mode called rebuild that rebuilds an RPM from a source RPM. Quality checking RPMs. During the first lab you install the RPM Lint command. That is a linter that will check spec file, source RPMs and RPMs. It'll identify real common packaging errors. It's not mandatory to resolve all of those errors and warnings, but it's good to try. But it does flag real common mistakes. But just to understand that it's not always gonna be possible. During the lab, some of the first examples don't have a man page with the program that you're gonna package. And you'll get a warning in RPM Lint that says warning no man page. Don't freak out, that's normal. It just happens sometimes. But that is a good indicator. If you're packaging some real software on the internet and it tells you, hey, no man page, you might think, does the software have a man page? And should I be including it? They do, let me add that. Or I checked and they don't. I'm gonna file an issue and ask them to write a man page or write the man page myself and contribute it upstream. So along with the RPM Lint, you can also use the RPM command itself for querying packages. Most of the time when you're doing packaging work, you're not gonna have these packages installed directly on your system. It's real common for package maintainers to be targeting a different operating system than what they're building on. And we'll talk about that in the last lab with a real cool tool called Mock. But you can use, for an uninstalled package, you can use the RPM command, use the package, dash, package flag, and that'll let you query information. A lot of people might have done like RPM-QI on an installed package before to show you the information about a package you have installed. You can do the exact same thing on a local package file that hasn't been installed by using the dash, dash, package flag. Some other flags that go along real good with that for inspection, dash, dash, info, list, your requires, provides. Those are gonna be dumping out the relevant type of metadata that you're asking for on the command line there. So let's get into the next lab. More hands-on, so I'm stopped not droning on as much.  That is just a dorky little name for a Hello World program written in Bash. Go ahead and click start on your lab, and I know the people that came in late. If y'all wanna join us, I'll show you the URL, but for over and else, go ahead and get started on the Bellow lab, and then stop whenever you hit the next button after you click next. So people that are just coming in, not to call y'all for being late to class, it's fine. But if you wanna join in on the workshop, this URL here will take you to the hands-on lab if you have a laptop to work on it. It'll be up after the workshop, so if you just need to write it down and do it again later, you can. It'll walk you through all these labs. There's lecture that goes along with it, but there should be enough instructions there along the side that you can get caught up if you want to. And if you have questions, if you're doing the lab and have questions, just raise your hand. Somebody with a mic, don't need a mic, somebody will come around and help you. I wanna let y'all know that we're on a little bit more of a time crunch than I expected originally due to some great questions that I don't mind at all, just over my minimum budget for time. So I'm gonna talk a little bit more about another packaging thing that we need for the next lab, and then open up the next lab so anyone that has already done with the Bellow lab can start on the next lab. If you're still working on Bellow, just kinda listen with one ear. This is a fairly, the next lab's fairly simple, and then get to the next lab when you get to it on your time. If you run out of time doing the lab, you're not really gonna run out of time. The scale folks told me that there's nobody in this room after us so we can actually go over and keep working, or if you just have to get somewhere else and want to write down that URL and finish the lab another time, that's also an option. But keep working on Bellow if you're still on that lab. But I'm gonna tell you about something that you'll need for the next lab, which is installing build requirements. For Bellow, there were none. It's just a bash script. Everything that's in the default environment is all you need to just get that file copied and then you're done. RPM build, it needs all of the build requirements that are listed in the spec file to be installed on your build host. You can do that either manually, just ahead of time, just install those by hand one by one. But if there's a lot of them, it's a lot easier to use a command called DNF build depth. And you will see that directly in the cello lab with an example of the exact usage. So I don't have to walk you through it. So when you finish the Bellow lab, click next and go on to the cello lab, which is appropriately a Hello World program written in C. And actually, there's no more lecture before the third lab, which is also a Hello World program, but written in Python. So just move through those labs as you get time. Do not feel like I'm rushing you. If you have questions, just raise your hand and we'll come by and get you on track. Don't get too bogged down if something's not making sense. And then there'll be one little last piece of, I'll give that a little bit of time to not overload you.  We'll save that for like the last 10 minutes. When everyone's head down, working through the labs, going smoothly. Here, I'll give you a couple more minutes, just working freehand.  So one more slide of lecture to tell you about a really cool thing. So during all these labs, y'all have been using the RPM build command.  I'm halfway kidding, but there are drawbacks to using RPM build directly. I mentioned that with the DNF buildup thing, your build requirements have to be installed on your host machine, on your build host. It's also really easy that if you have a build requirement that is already installed, to forget to list it as a build requirement. Like if you have, like if your software needs GCC to build the package and you have GCC installed for some other reason and you build your package and you don't list build requires GCC, somebody else that tries to build it might fail to build because you didn't list it as a build requires. So it's entirely too easy to get in that situation whenever using RPM build directly. You can also only build RPMs targeting the same operating system and operating system version as your build host. So your build host there in that lab, it is a CentOS Stream 9 system. And so you really can only build RPMs that officially target that. You might be able to make an RPM that accidentally installs on another version or on another distro like Fedora, but it's not guaranteed and you'll probably run into library stuff eventually that for not targeting correctly. But there is a solution for that. And it's called Mock. And that lets you build RPMs in isolated charoots. It still uses RPM build internally in its guts, but it gives you the benefit that your build requirements are not installed on your host, they're installed inside that charoot and then discarded when you're done with the build. That'll help you identify any missing build requirements because in that charoot, it is a very stripped down minimal, just the bare minimum structure for getting a build done. And it'll tell you like, I also need these development libraries or this compiler installed to complete the build. Even more importantly than that, that's a great benefit on its own, but it also lets you build RPMs that target different operating systems or operating system versions as the build host. Limited, right? Like there's some limit of support for doing different architectures, but you have to have cross-compile support, I think. Okay, so that's gotten better then. There are also ways you can use Mock to do architecture, different architectures, but that's, yeah, you probably won't be doing that. Yeah. Those charoots that it makes are also automatically created and removed and managed so you don't have to really think about it. It's just, you might end up with a fairly large varlib Mock directory over time that sometimes you might want to purge manually. Mock is really widely used in our EGIS system. It's the Koji build system that Fedora uses. Uses Mock inside of it. The copper build system in Fedora also uses Mock, so they have a standard back-end, essentially. The Fedora Fed package tooling, if you get into being a Fedora packageer, it also integrates really well with Mock. And that's what you're gonna learn about. In the next lab, you're gonna build the Pelo package again but you're gonna use Mock to do it and instead of building it for EL9, the same as your build host, you're actually build it to target EL8 instead and then inspect the results. Enterprise Linux, and the answer is yes. Sent to a stream is the reference implementation of Enterprise Linux. Owen, we're close to the end of our official time. Keep working on the lab if you need to. I'll still be here for, I'll basically be here till they kick us out. But one other thing I wanna bring up is if this was interesting to you and you enjoyed these labs, then I would encourage you to consider becoming a Fedora slash Apple packageer. This URL here, bit.ly slash Fedora packageer, that'll take you to a page in the Fedora documentation about becoming a packageer. Coincidentally, that is great background for getting involved contributing to CentOS as well because it's a lot of overlapping these ecosystems, these communities and the tooling, the tooling's mostly the same across and it's all relevant and connected. So, if this, like me, if it scratches an inch in your brain where this was interesting and exciting to you, check this out. And I hope to see you around Fedora and Apple and CentOS spaces. That's basically the end of the stuff, but keep working on the labs. I don't know if Sean wants to say last words for the overall thing. No, not last words. I just wanted to, like if you have, you take this back and you then have more questions and you wanna find Carl, there is a CentOS booth, exhibit hall opens tomorrow afternoon, I think or whatever, so there's a CentOS booth. We'll be there. Yeah, there's also a Fedora booth that you can go visit and people there know stuff too. So, both a Fedora and a CentOS booth and a Red Hat booth apparently. You don't have to get out of here.  No, I'm just kidding. I'm gonna be here till they kick us out. If you wanna keep doing the labs, feel free. We ran a little over on some of the lecture, my fault. So, we will be available and or write down those URLs. It was bit.ly, hello RPM and Fedora Packager were the two URLs to get to the hands-on lab and to become a Packager documentation. You want those again? Yeah, I'll be here. If you have more questions on the lab, just raise your hand.