Before we actually get started, a question. Who here has their background set up? Like, who here has followed the instructions in the wiki and they have a virtual machine running for the demo? Oh, a few people. Congratulations, you all passed. The rest of you, how many of you have Mac.RM1s or two? Okay, first of all, how dare you come to a Linux conference. But the second thing, the setup will not work for you. Because Virtual Vox has decided not to support M1 and M2. But for you, we have a solution which also can apply to everybody else. We have some Linux instances running. So, I strongly recommend that you set up your background set up. It's pretty easy.  But if you don't want to go through that hustle and you want to make me spend money, but it's not that much money, Linux is a good sponsor. Not sponsored, if you know. You can go to that URL or that QR code. And if we did this correctly, it will take you to this. Hopefully yours will not be in Spanish. So, these are Linux instances. You pick one, put your name or your next handle or signature or anything next to the one. And that's yours for the entire workshop. I use a Sage like that, that's the password. It machines, I will kill it as soon as we finish the tutorial, so you cannot mind Bitcoin between, you can mind Bitcoin between now and the end of the conference. So, I'm gonna come back to the mic in about like four to five minutes and ask the same question again. And if 80% of the hands are up, we go start it. All right, the question was can I put the URL up? And the follow-up question was can I make it bigger? There you go. But again, please do the background, it's better for you. If anyone needs help, you can raise your hand and one of us will come over and attempt to help you. Attempt to be in the keyword. Yeah, no guarantees. For people who just arrived, we are doing setup right now, or you are doing setup, we're just watching you do setup. If you are able to, if you're running Linux, or are able to use the vagrant instructions to do the workshop locally, just go straight to the GitHub and go to the workshop, read me and follow those instructions. If you're on Mac, how dare you? Just scan the QR code and reserve one of the Linode instances so you can do the workshop remotely. And raise your hand if you need help and we'll come over. So, but before you raise your hand for help, who here has their setup ready by now? Either SSH into a host. Okay, perfect.  Okay, so we are gonna get started. Okay, so welcome to this workshop. Do you want to introduce the workshop? Okay, so in case you didn't know, this is system D101. This is not a rehash of the workshop that we did last year, even though a lot of it is similar. But we decided to split the workshop of last year and added more content, so we split it into two sections, system D basics and system D advanced, right? So today's basics. Hopefully you will all be here tomorrow for system D advance as traditional. There will be snacks in the middle of this workshop, so if you stay, you can get to eat some delicious food. Yeah, and those are the ground rules. My name is Alvaro, this is Anita. We both work at Meta, production engineer, software engineering manager, but this talks has nothing to do with Meta. There's nothing in it that I would say it's particular to Meta, and we did it that like on purpose. So this is all information that you can take home and use it on your own machines. So before we had started, who here is kind of familiar with system D? Okay, that's great. There you go. So this is probably, there's gonna be some sections of system D that it's gonna be a surprise for all, for some of you, but it's good to have people who know about it. Well, I guess we're in a Linux conference, like everybody should kind of be familiar with it. I'm assuming that this part is not recorded, so who here is leaning positive towards system D in their general sentiment? Okay, good. Who here is leaning negative towards system D in the general sentiment? Okay, perfect, and I assume that everybody is in the middle. The thing starts so far, so it's good enough for me, right? All right, cool. So let's get started. We're gonna start with a little bit of a small slide. There's no need to need it, and there's no need to actually follow the slides. After this slide, we're gonna do the we're gonna start the workshop, okay? Do you wanna start this way? Oh gosh, what is a service manager? Well, I mean system D, the service manager, yes. Oh gosh, boo. Should we get, we can do that in it. Yeah, we'll call A.V. back later. In the meantime, I will try to use speaker. The speaker seems to not be working. State, right, not working. Yeah, okay. I'm just gonna keep going. So you'll hear us say service manager a lot. Basically, we're just referring to something that manages your services, or in this case, just the processes on your hosts. It allows you to find some operations for your services, and also provide some guarantees about life cycle in theory. In the good old days, we had these init scripts, and that was our idea of service manager.  You have to kind of define all of the verbiage for how you want your service to actually operate.  The problem with this is it's not really clear how you start and stop a method. It's not exactly the same between each script. So for example, in this Cassandra example, are there like two start commands? I don't know. Yeah, for instance, if you were to ask on the previous command, what is the user that start Cassandra, it would not be clear unless you're actually familiar with all of those commands, right?  So the slides, no, and these are gonna be very short. But after this, we're gonna go to the GitHub. And there is a GitHub where we do the tutorial. Sorry, the question was, is there a website where we can follow the slides? Go ahead. And maybe even more importantly, there is no way to figure out whether your service was actually running and getting kind of observability or logs out of it. At least not in a standardized way. So we kind of named those problems, but additionally, boot and shutdown was slow. Like I said, getting logs was not standardized, so debugging was kind of hard. And at the time, Upstart and SystemD were emerging technologies. And in the end, SystemD won. So that's what the majority of Linux distribution these days are now using to manage their services. Yeah, so this is just ceiling into the territory of the workshop, but the main difference between, and you're gonna see it when we start, the main difference between SystemD and Unity is that in your batch script that you were doing, you tell the Unity what to do and how to do it. While in SystemD, you tell just what to do. SystemD will figure out how to do it, what is the best way to represent the things that you want. Okay, so now the workshop, if you wanna follow along, is on this URL. I'm gonna give you all a second to go into the URL and then we'll get started. We can get the URL back up. So the way this workshop is written is that you can actually do most of this on your own time if you just, if you fall behind or you're not able to get your setup or anything. You can watch us do it and then you can also take this workshop home and like do it and follow the instructions. If during the workshop you need help or you get stuck somewhere, just raise your hand. One of us who's not speaking might come over, might, if you have a question though, just to distinguish between whether you need help or not, please raise two hands, just so we know to call on you and answer your question. Okay, so the, if all of you heard the background, the way that you start this thing is you do background up and this will pull the latest background that we have built that I built yesterday.  The question was kind of doing more and the answer is yes. Is that better? Perfect. Okay, cool, so there's three. Okay, we still have some Linux instances available and if they run out, we can just provision some on the fly. So feel free to use them. Okay, cool, so once you do background up, background SSH should take you inside the machine and this is a, hello, what we, this is a virtual machine running on your host. It's run Fedora 39, if I'm not mistaken. It has the latest and brightest of system D and has a lot of code examples already made for you. So we start with the, so what you do on GitHub, you navigate to workshop, getting started and then the read me and this is exactly what we went through. So we can get started with system D. Okay, that sounds like I can maybe make this work. Okay, cool, so what we are gonna go with, what we're gonna start is basically the most simple part of system D which is creating system units. All right, so what I ask people to do in this one is that in your terminal, in your, you can go to ETC system, you can do LS, so you can do LS and then do a follicle ETC system D systems and then you're gonna see that right there, there is a bunch of files that has extensions. Some of these extensions are target, some extensions are service, there's gonna be timers and other things. Those are the ones that we're gonna go and check out between today and tomorrow. So the first things that we probably wanna do is that we want to create a file that is called My First Service. So you can use PIM if you're comfortable with it and then just copy the content of this into the file. You can use the Nano, I think we have Nano on maybe Emax. So this should be the simplest service unit that you can create. It's still a little bit more complicated than the simplest simplest, but it gives you like a good understanding of actually how you talk to system D. So here you see you have a declarative document instead of having a shell script that define how to start your application or where to start your pit files and all the things. You have a declarative file that tells you the description, what is your file, right? What is the service that you're creating? It has an application that you use to run, has a policy for how the service is gonna, sorry, it has a policy for how the service is gonna restart and has an install section that we're gonna definitely look at it later, but it basically, if you remember the Google run levels, those are your install. So if somebody wanna install everything that has to do with users, all they need to do is start multi-user target and this unit becomes part of that, right? So there's no longer they need to put this into a particular folder, you can just describe what you're doing. This, though, assume that there is a script here that is called service whoamai.py and it's executable, right? So let's see if this is true and you can go just copy and paste it and execute it. And this very little clown script, like everything that I write, is just display a bunch of information about the running process, right? It has the infamous Python syntax warning. It shows you who started the unit, the scope, whatever that means. Ask what is the main bit, whatever that means. And then it just print a bunch of environmental variable that as you can see, those are, as Unix always do, inherent from my environment, right? So if I type M, you see that those environmental variables seems pretty similar. By the way, if you are using background or the keynote and then you're in a Tmax shell, which you probably should, if you press Ctrl K, you can clean your buffer. So it's a good thing to have image clean. Okay, so now we have the service. So we have tell system D what it has to do, which is how to start the unit. Now we need to tell it to start it, right? So and the way that it works is that you use system CTL, you do start, and then you specify the name of the thing that you just created. My first service of service and you started and very anti-climatic, nothing really happens, just returns. So yeah, what happened? We can go and change this and ask for status. And I know that today we take this for granted, but the fact that we have so much information about how a service is running for free by just saying this is the script that you used to, like this is what I want to start, it is pretty amazing.  First of all, it tells you where the service is defined. Very useful. It tells you what is the status. It's running, it start running 18 seconds ago. It tells you a main feed. Okay, so this is interesting. So now you have a unit and there's something in that unit that is the root of that unit, that is the main feed. It tells you how many tasks it has. Task is basically how much forks are running. So when you start an application and then you wanna like to allow to do something, you need to fork an exec. Each fork counts as a task. It tells you how much CPU is using for whatever that means. And it has this beautiful, beautiful thing over here that it tells you C group. So before we kind of dive into this, and we have a tutorial for C groups here, how many people here are familiar with C groups? Okay, nice. This is great because like people know system D, but now they, so they might not learn much, but now they are gonna learn a lot about C groups. So before we actually talk about what our C groups are, just imagine that C group is a control group and it's a way for you to impose restrictions on a service. But system D does something very interesting with C groups. It is for each service that it starts, it put all of those, all of the things that are on the service on a C group per service, right? So this one has a C group that is called myfirstservice.service, right? So it basically, when you say I wanna start my service, it creates a C group and then everything that starts goes in there. So now system D can keep track in a better way of what things started with the service, right? So in the good old days, when you start Apache, who I'm all, and you wanna like stop Apache, what you do is that you kill the main pit, right? You have to figure out because Apache would write it like a pit file. But you never really knew what did Apache started, right? So like, sometimes you chill out and then you double forks or you escape the parent. So there's no way of keeping track of that.  With system D that doesn't happen because system D can keep track of all the services because it keeps tracks of the C group. And when you say stop the unit, it will stop the C group. So yeah, so let's do that. I believe that's the 10 things that is, let's stop the service. Oh, by the way, sorry. And here in this, you can also see there is a little bit of the output of the unit, which is also kind of useful. Oh, sorry, and the unit just sleep infinity. I will probably show you what the unit does, but later. So okay, let's, we can stop the unit. So to stop the unit, we do something very simple with if you do start for start, you do stop for stop. That makes perfect sense. So now let's go back and do status. And now you see that it provided you all the information that you need, right? So it looks correctly that system D was the one to stop the unit where the unit was. What is the status? So it gives you all this information that you may want to have from the unit. Now, if you want this to actually start the service when the system start, you need to enable the unit. And if we go here and we do system CTL enable, you're gonna see that a system D create a sim link for the service units that you just wrote, and it put it there into multi-user target. Not to spoil the future, but what is gonna happen eventually when you start your machine, system D will start the first target that start all the targets. And one of those targets is multi-user. And when system D start multi-user, it will start all the units that were installed by multi-user. I will do that in parallel as much as it can. So it makes boot fast. So so far, everybody who's following along or there's somebody who is behind it, or are following along. Perfect. Success, we did it. Okay, cool. So let's move on to modifying a unit, right? Again, this is the system D one-on-one. So we have to go through all the basics so we can do better. All right, so let's do something very interesting. We can take the unit that we need and we just modify that we just created over here. In case if you're typing along and you don't know how to do this, you can do control R to the reverse search. And then you find BIM, which was the command that I use. Presenter, it takes you there. And then we're gonna add here the user, nobody, right? So in Linux, there's basically only two users that are present in every single host. So the first one is root, the second one is nobody. That is kind of like the only one that systems tends to agree on and then everything else can even have like different UIDs. So we are gonna start, so we're gonna modify this unit at the user, nobody. And now by looking at the unit, you can actually see what user it's gonna run the service. That's what we wanna do, right? All right, cool. So let's start the unit, okay? I'm gonna take some time. I'm gonna start it. And then a system D did start the unit. Like if I do status, you're gonna see that it start the unit, but it actually complain about something, right? So it says like the unit file or drop-ins have changed on disk, but system D will not pick up those changes unless you explicitly tell it to pick those changes, right? This is mostly because you care about atomic changes. Like sometimes you need to modify more than one unit at the same time. So if you modify one and those changes get picked up immediately, like catastrophic things can happen, right? So system D allows you to like reload everything when you start.  And now you do this little command over here. Systemctl.demon reload. You do that and it holds. What basically that does, system D will go and read every single unit. And now you can start and nobody's complaining. Now the beauty of this is that, remember this old status? For some reason I thought the user would have been the status, but yeah, but it, but it, yeah. So, and it shows that the user that I was using is nobody. Okay. I'm gonna do, there you go. So let's take a look at this script. Service, who am I? The best way you can do this is you just cut the script. It's very short, but also you can see it because it should be on your bin folder. Service, who am I? Yes, sorry. Those are hands for questions. Okay. It's do everything at a change. It's atomic. Sorry. Oh, yes. Thank you very much. Yes. So the question was does system D, when you do the more reload, does it reload everything? Or you can specify certain things. It does everything at the same time. That was my understanding about a year or a year and a half ago. I don't know if that have changed. But yeah, it's atomic. I used to remember that this is like an event loop, right? So if, like, it actually has to like reload everything. So it has to stop. So this process tends to be very heavy. You don't wanna do that like every single time. Cool. Any other questions that people may have right now? Oh, this doesn't look good at all. Sorry. Does this one looks better? I think it does. Okay. So let's see a little bit about the script that we are running. We're gonna abuse Python a lot here. We do some coding in shell and in C to show you later, but in general we try to do most of our stuff in Python. So the script is pretty simple. It basically just figured out stuff about the unit. Then it prints this beautiful ASCII art. It tells you what is the user. It tells you what the way it was running. It tells you something, something, what is the service. And then it prints all the environmental variables. If there is no sleep, environmental variable, sorry, as long as there's no sleep environmental variable, what it does, it goes and exec into sleep. So whatever, if you saw here into the terminal when I did status, you see that the thing that is running there, it's sleep infinity and not the actual exec unit, right? So it keeps track exactly what it is. It kind of also tells you like the POD. So, and question because I didn't realize, do you guys see the, is okay, perfect. That seems to be better. I don't know if it's the best, but it's the best I can do. I go, so, oh yeah, okay, cool. So yeah, so we already went for what the script does. So let's do some journal CTL and let's learn a new command, right? So you can paste that journal CTL command. And this is actually quite interesting, right? So journal CTL, it's one thing that comes out of the box with system D. We have a full session on it. But for now, what it does is that everything that goes into STD out or STD here of your unit gets logs into the unit, right? So you get log for free. No need to start like skimming into bar log, myService.log, even though you can still do that. But the point is that system D journal, sorry, journal D will log your things, right? So now you can actually see the entire output. Remember the status has a little output at the end. Now this is like the entire output. And you can see that it shows like the informations that we were looking. Yeah, no need to know everything, but in general this just tells you show me the last 30 lines. No host name tells me don't show me the host name because that's boring. And then that's you tells you show me the unit. If you were to remove, let's do this just because of fun. If you were doing journal CTL.f as follow, as you would tell, and don't put any unit in it, you are gonna see that it's basically just showing you everything that happened on the system, right? That's not part of this tutorial, but I thought it was a fun thing to know. Okay, cool. So override units, all right, cool. So let's do some override units. So there's a few ways in which you can evade a system unit. It's a text file, so you can just beam into it and then, great, you're good. But maybe you want to do something that is called an override. And override is pretty useful, right?   So when you install the NGINX RPM, it comes with a system unit, which is the thing that the vendor tells you or tells how their system should start. You don't want to modify that unit, right? Because, like, if, first of all, if you want to change the user, that system, it starts the service. And you don't want to modify that unit because if that's managed by an RPM, when you reinstall, either the RPM will decide not to upgrade that version or worse, it will override it. RPM will not override it, but in general, like, every package manager can do that. So the way the system dissolves this problem is by allowing you to have override configs. So these are configs that you put next to your configs that allows you to extend it. Or, what is the opposite of extend it? Reduce, I don't know. Like, if you don't like the something start as a user and you don't like it and you want to remove that option, you can do it also with override. Hey, so let's do this. So the, okay, so the way that you would do this is that we're not gonna do it like this, but if you definitely just want to do this by hand, you create, remember your unit was called MyService.Service. So if you create a directory there called the name of your unit, dot t, and then add any conf inside of it, this is called an override unit, right?  But instead of doing that, we can actually do the thing that is called systemctl edit. Since systemd already knows where the files are and where it has to put the files, like, sure, it must be nice to help people out. So if you do systemctl edit, you get this beautiful thing, right? And it tells you, hey, I am doing exactly what you were planning to do by hand. I am editing a EDC systemd system, MyService.Service.T, override.conf, right? And it kind of shows you the previous unit. So this still exists, this is the other unit, whatever you put here, it is what it is. So, what are we doing? Oh, yes, we are adding an environmental variable. So we don't have to put everything again. We don't have to specify everything, we're just gonna say for service, the environment, add special m equal equals activated. I think we can just save it and then it tells you successfully install. So, oh, sorry, there was a question, Anita. You probably know this answer better than me and I don't. When you do demon reload, systemd reloads all the units, right? But can you reload just the unit that you modified? No, you, just the content itself, you can't do individual unit reloads. Yeah, I did remember that, yes. So, cool, I didn't lie to you, that's good. All right, so, that's the one. All right, so let's do, so we modified the unit, we added stuff, so let's restart the unit, perfect. By the way, when we did systemd still edit, systemd also take care of the heavy lifting of not only finding where the unit is, writing the override, saving it, it also reload the demon, so you don't need to do demon reload if you do it this way. So, I restarted and then we're gonna do this journal CTL thing over here. So, I know we're dumping a lot of syntax to you, but N, like we say, is 15 lines, the unit, and we're doing G for grep. So, we're grep-ing this, a content of this line, right? So, if I do that, it kind of shows me that the unit with that POD has the special M activated, which is the thing that we added. So, great, it works. All right, so, cool, so let's do, let's modify the unit again, and now you can see that we have the previous unit. Let's add this memory max over here, that is 10 megabytes, right? So, we're just setting one of many systemd options to control how much memory the service can use. We save it and then we can do the following, and this is why I wanted to do this, I remember now. Now, you can see that there is an override, right? And then, that the memory here is set directly to max, right? So, I want to explain, like, why do I think this example is useful? Remember that we changed the user, and we needed, or we added a manager variable, and we needed to restart the unit for that change to pick up, right? But now, I changed another setting, which was the memory, and I didn't need to restart the unit for it to pick up. So, anybody here can say, why do you think this happens, besides making life inconsistent? Yes, go ahead, anybody, just shout it. Oh yeah, but we did also this with an override file, right? So, the thing is that even it has to be something on the option, you wanted to say something.  Sorry, I didn't hear you. Do you see that a running process needs more memory? You can give it more memory without needing to restart it? Yes, so it is that. I will repeat in case that isn't sound. She said that if the process needs more memory, you can just give it more memory, you don't need to restart it, and that is true, right? So, like, systemD will try to do its best not to restart service, not to need that. And in order for the restrict memory, what you do is that you modify the C group, which is something that you can do live, right? While adding an environmental variable, you can only do it at the startup of the application, right? So, those are kind of things, right? So, in a way, this does obstructs you a lot of trying to understand what's going on under the hood, but the point is that if you do it with a bit, systemD will take care of doing most of the heavy listing for you. Okay, so that was it. And yeah, and this is, we hit, go ahead, two hands, good. The question was, can I talk about some cases for the systemD overrides? So, the one that I give about NGINX, it's kind of pretty common in terms of vendors, right? NGINX is kind of the default application that my brain goes, but in general, like if you're a vendor and you want to do applications, that's what you do. The other thing is that if you're a system administrator, one of the things that you tend to do, you also add certain overrides, so that every application that runs on your host like has a firewall, which is an option that you can add, right? Or you have a, or you run something before every unit starts, there's an option that is called pre-start or pre-exec-start or something like that, that is something that start before your application starts, right? So you can add something to log, I don't know, right? So those are the kind of things when you want to do modifications on the unit without actually touching the unit. Do you have another example of that, Anita? Isn't the one that comes home with Fedora? Yes. All right, so Fedora comes with an override. What does that override does? The one that changes the abort and timeout behavior on units, that's supposed to speed up startup. I mostly use overrides to override vendor-shipped units, like if they ship, if they have something for SSHD to service and I don't like a certain behavior, I would use an override for that. It just really allows you a way to further tweak what's already on the system. Yeah, the Fedora one that you were saying is that like the Fedora, like the Fedora pack? Yeah, Fedora also use ships overrides for their units as well, or like certain packages units. Yes, okay, cool. So the rest of this part of the tutorial are us just going through a bunch of systemd commands. It should go pretty fast, and then we're gonna take our first break because it's unfair to keep three people like locked up for three hours. And then you get to hear Anita instead of me, so that's a plus. All right, cool, so let's, okay, so this is quick and dirty walkthroughs to some systemd commands. We already went to status and then the unit name. You already saw that. What you didn't saw is that you can do status and you don't need to put a unit name, right? So if you just do status, just like that, it shows you a kind of a list of the system from the point of view of systemd, right? So basically you're asking for the status of P1. So, sorry, not the status of P1, that's not what it is, but it kind of give you like, oh, so these are all the things that are running, all the processes, kind of like give you like a nice overview. This is a virtual machine, it has very little going on. So that's cool. Now, a lot of the time you do PS outs and then you see all the processes, you don't know what's the service of a unit. Like if I do PS outs F, you're gonna see that list. And yeah, there's a slip's finity, but I have no idea what's the unit that that is. So I cannot ask for the status of that unit, but what I can do is I can ask status for 5423, which is a magic number. No, I'm lying. It's a PYD of the unit, right? So if you just know, sorry, of the process, if you just know the PYD, system D will, since it's keeping track of everything that is under the hood, you can use system D status by the PYD and that's all good. If you do system CTL without the status, you get a horrible view, but it does tells you like all the units that were executed in the service. Or in the machine. So it has a lot of information and if one of those is read, it means that it failed. Nothing failed here. Okay, so that's status. Very nice, very useful. Now, assuming now that you do know your service, like this one, and you want to actually see the content of your service, you can do system CTL cut on the name of the service. Again, since systems D knows exactly where everything is, it can nicely assemble the list of files for you and display it. Very useful. And it displays exactly what we say it should display. Now, if you don't want it to cut, if you want it to show, whatever that means, nah, we know what it is. There, it will kind of tells you all the properties that the unit has set. All right, so these are all the options. Go ahead. Yes, yes, yes, yes. So you definitely can do, so the thing is system D keeps the status of the entire, so the question was if we can use this to pick what you can put in override and the answer is yes. System D keep a list of all the things that are running. And sorry, and for each service, it has a full list of properties. So if you set ones, the other ones are the default value, right? So now this will show you all everything, so you can override whatever you want. With that said, if I'm not mistaken, this one is using the divas interface. So we're gonna go that. So this doesn't really map one to one to the actual setting. Like for instance, here it says restart you, I'm not sure. Yeah, so it says restart you seconds. All right, let me go to the terminal. Restart max you second, that's a microseconds, I believe. It's, yes, it's microseconds. The unit, the settings is restart max seconds, right? So you don't use the use seconds there or runtime max or time or something like that. You can check the docs, but that's nano seconds, nano, nano, okay, sure, right? Max is hard, right? Oh yes, no, no, no, perfect. But yeah, that you can do. You can also expect just a few settings there. So you don't need to watch everything. You can just specify that to be and it will tell you what it is. And here's what I was telling you, right? So you have exec start and it kind of show you the way that system sees this option, right? Like with all the different properties. But it's kind of like, if you wanna know the main PID, you can do this main PID, but do a specify the unit. And this is very useful also for Chelscript. So if you specify like that's P main PID, it will tell you main PID equals on the PID. You can also do, add this option over here that is called value. And then it will just give you the value.  Any question, comments so far? We're, yes, you're good. Chris, interrupting. So the thing with system CTL show is that it only has what system D is stored in memory at this moment. And that's not necessarily the same as what system D is already running. So certain properties, once the unit is running already, you know, that's it. But you can reload and change the unit and system CTL show will show what system D has in memory. And you know, it's not exactly the same. So just keep that in mind when you're looking at show and you're confused like, this says this one setting, but you know, when I'm actually running the service, it says something else. That's why. Yes, question. Yeah, so the question here, it's the like, I want to paraphrase you pick at the word watchdog and ask if you can set the watchdog for service. And the answer is yes, you can. And we actually have a tutorial for that, which comes tomorrow. Yeah, so many things you can do. Go ahead. The question was, when you use system D kill is when the service doesn't respond? So no. So if you do system CTL stop, like if your service is unresponsive, you can still use system CTL stop.    So the stop is supposed to kill the process at the end of the thing. Very rarely fails.   System CTL kill, you can do that, but you don't need to know the PYD, you just need to know the service name. So you can say kill the service and sick hub to NGINX and then system D will figure it out what's the right way to do this. So with that, we're gonna do system CTL kill. And yeah, it is exactly that, right? So you send the signal, ah yeah, but this is actually quite interesting.  And then I'm gonna do status, that's not the status that I want, that's the status. Wait, did I send the signal? Oh, oh great, sorry. Yeah, so you can see like the unit's still running, but the startup time is seven seconds ago, right? So what happened there is that I send the sick kill, like system D send the sick kill to the proper main PIV, that thing died, but the system D has the setting that you remember that it's just like restart always, that we put it, so the unit restarted, right? So that's kind of like intended. If I were to do stop and then do status, you see that the unit is stop. Is it stop? Yes, it's stop, all right, there it says, right? So the difference, so the difference was that system CTL kill, after that, the output is not that the unit has to remain stop.  But it's very useful.  Yes, it's the unit's kill, but also the unit's kill has a very bad name. Okay, so there is a system CTL this unit, which is the same one that we already, so when you call system CTL with no parameters, it's exactly the same as calling it with no units. So it would list units. So this one, it's exactly the same as this one, and that again shows you like a nice view of everything that has run on the service. While system CTL status shows you everything that is running on the service, right? While the other ones shows you like things like these weird devices that are for mounting your TTWise, that will not appear otherwise. Did you add this or did I add this? I didn't remember this. Did we add this, did you add this? Okay, cool, we'll pass Alvaro. You have list units, so it's an awesome thing. You can use wildcards for searching, you're basically saying like give me all the service, you can do that. It's also very useful. Yeah, it also works with system CTL status, like it's the wildcard, right? So it kind of shows you like everything that has a service. You can be more specific than just service, but that's it. And I think there's other things that are cool, it's called list pass, list socket, list timers, that do all the things. And it's 11.04. I think it's a good time for us to take at least 10 minutes, bathroom break, coffee break. Thank you very much. If somebody have questions? Oh yeah, oh yeah. So is it time for portals? Yeah, and your prize for being, your reward for being here and learning about system D is that we have some portals pastries here. It's time, okay perfect, niche approved. So yeah, so you can come and grab some portals pastries, they're free.  Thank you, by the way. Okay, four minutes and we'll get back to these fun times. Great town, it is, yeah. I was scammed out of coffee, there's no coffee. Okay, one, two, check. Okay, we are ready to get started. So if people can take your seats in a color. Okay, so for the last part of system D 101, we're gonna talk about ephemeral services.  The only downside with this is that you, it doesn't preserve the unit between like restarts and things like that, it's just stored in memory. So we can give that a try by running this command here, system D run, service who am I pie, it's the same one we've been using before. So now you see that system D created a unit file and it's starting to run it. We can take a look at the contents of this unit by copying the unit name and doing system CTL cat. And you can see that system D stores the unit file in slash run, which is in memory. And it has the contents of what it generated here. So it should look very familiar to you already since you created a basic unit at first. The other thing you can do with system D run is create an interactive shell. So if you do system D run dash dash shell, this will kind of drop you into the shell inside of the system D unit. So you can do system CTL status to get what the pit of this unit is. And you'll see that from within the shell in the system D run unit, it looks kind of like this. You can go ahead and play around with this for a bit, but it's basically just another bash shell within a bash shell. And then when you're done, you just go ahead and exit and you'll be back on the host or the virtual machine wherever you are. Any questions about that? The question was why might you want to do that? Do you mean system D run or system D run shell? Okay, so one of the useful things about system D run shell is that if you're debugging some kind of complex behavior with your unit, you can use it to kind of drop into what the state of system D is like. For example, hmm, you got an example, Alvaro? Yes, so well, you can use this for tutorials and show how things work. But in general, it's kind of like looking at the world from the point of view of the unit, right? So see, later we're gonna see that system D can do things with your file system so the unit may sync that route is read-only, right? So you may not really know that that becomes a problem so you can start a shell with the route as read-only and then do the commands that your application would do and that gives you like a nice feeling of how the unit works. That is kind of very useful. But we use it extensively in this tutorial because we ask you modify this. Now go into a shell and run a few commands and see what effect that has in the world. I'd say like just system D run shell by itself may not be super special but once you start getting into the sandboxing properties and you run another process along with the shell or something, you'll be like, hey, why isn't this running properly compared to outside of the host and it's because of how the system D property is behaving. So it kind of allows you to test some of those in more detail as well. Any other questions? So the question was how does system D know to drop into that particular shell? I mean, it's creating the unit at the same time as it's dropping you into the shell or like it creates it, run execs and then you're like in the shell there. It connects via the pseudo terminal there. Does that make sense? Yes. This becomes more clear maybe in the second example. So we're gonna start using system D run shell more extensively to run an interactive process. So the system D run shell is pretty much the same as adding this dash dash PTY argument. So you can do more fancy things like I was saying by copying this command and we add dynamic user true, which is one of the properties that allows you to run the process as a random user. So let's go ahead and do that now. So let's go ahead and do that now. Sorry. So we do that. You'll see now that the service who am I, Python script that we ran before is now running as a kind of random user that system D has generated. Later in the workshop we'll talk more about how dynamic user works, but just know that it's one of the ways you can use it to run your process as an arbitrary user. For the very first basics of system D. Are there any questions about this part before we move to the next part of the workshop? So now we're gonna start talking about other system D unit types. So far we've pretty much only talked about service units because those are the ones that allow you to actually run commands and execute processes. The system D has a whole suite of other unit types that allow you to do other things. So for example, timers. If you are familiar with the cron tab, it allows you to kind of periodically run an executable or something on a schedule. And so if you recall the cron format, if you run something every 10th minute, you can do it like this and then you add the script or whatever at the end of the line. To do that with system D, you use a timer unit. So we can go ahead and create a timer unit with these contents. We're gonna use system CTL edit so that we don't have to worry about reloading the unit later. So the difference between the service is this timer section here. And the property, one of the properties under timers is on calendar and that allows you to kind of insert a cron style format and allow you to run the timer. And if we want this to run at, or like start the timer at boot, we also added this install section and it's wanted by timers.target. Let's go ahead and exit. Because we use the system CTL edit, it already reloaded for us. So that set up the timer section. How do we actually include the script that it's supposed to run? We do that with, by creating a service unit of the same name. And we can copy these contents here. And this is just a generic service unit like what we saw before in the previous tutorial. What this is gonna do is run service who am I, a Python script you're already familiar with and set the environment variable to no sleep equals no. Let's go ahead and save this as well. And now we can actually, we can go ahead and start the timer and watch when it fires. So I'm gonna go ahead and copy this, start the timer and we use watch here to see when the status changes on either the timer or the service unit. So you saw that the timer executed and the service status now spits out the usual logs that you're familiar with. Yes, the other thing we're noting about the timer is the, yes, let's dig into the status of the timer unit a bit more. Delete this watch command. So if you're already familiar with system CTL status, on timers we also get this trigger and triggers behavior. So what that does is it will tell you when the timer will next run and trigger this service. There is a property where you can actually change the service that the timer triggers. You can kind of look that up in the man page, but by default if the service has the same name as the timer it will trigger that service. What else is interesting here? We already know about loaded, it's when system D had loaded this unit into memory. Yes, question. So the question is what's the advantage of using system D timers over the previous cron jobs? So using on calendar we don't get to see a lot like the special sauce with timers. We're gonna go into some more properties later. But I think the primary advantage is that we can actually trigger a whole service file. And with that you can define, you can fine tune the behavior of how the command should run in relation to the timer. Whereas with cron tab it just kind of runs the script and then it exits. So yeah, so that's that one. There's other two things that I think I kind of like mainly superior to cron, but first let's say cron it is very simple and it is very familiar. So you can use that one if you like, right? But yeah, so the first one is the use of time. System D has monotonic, so you can say trigger these five seconds after the previous execution. Or you can do, since boot, do this half an hour after boot, like those kind of things. It doesn't have to be tied to a particular time of day. There's also flexibility in giving you supplies. I don't know if that's the right word, but it is like start this at one plus minus five seconds. So you can distribute a whole set of things in time. So that's the first one that I would say it's pretty useful. And the other one, it is where the strong suit of System D is that you actually get status and locks. And like in cron it's really hard to see if your thing run and what was the status of that run. You need to spec locks. In System D you just say like status of my timer and it will tell you yes, I was here. I did what I was supposed to do and it exit successfully our failure, right? So it cannot give you that. Well now we're gonna see all the things that Alvaro just described. So starting with the monotonic timers, let's just go ahead and edit our first timer and change on calendar to instead use on active 30 seconds. So what on active second does is it basically says let's just run this or execute this timer 30 seconds after we activate the unit. And basically it will never run again. So you can also combine this behavior with, that by itself is probably not super interesting but you can also combine it with like on calendar, on active seconds, on boot and things like that to further change how you wanna start the unit behavior. So let's start the on active. So if you're looking at this watch, you see that there is a countdown till the unit's gonna trigger. Oh my God. Great, it triggered. And in theory if we look at it again it will not show like a next trigger time. Yeah, it elapsed and that was pretty much it. Their system has a bunch of different properties like that. In the next example we have on active seconds, on boot seconds and probably a few more that I'm forgetting but you can always look in the man page for the rest. Okay, so similar to ephemeral service units you can also create ephemeral timers if you just kind of wanna run something on the fly. So if you do system D run on active 30 seconds and just echo this, echo hello from the past, it will automatically create a timer and the corresponding service unit for you. As seen here you see that it's now running this timer unit and also will run this service unit when it fires. And similar to before you can do and you'll see that the timer is already active and it's just waiting to, well it probably already ran because I didn't do the watch. It already created the service and the timer and it was gonna fire it after the 30 seconds. Yes, and thanks to Alvaro we can see that, well it was active 39 seconds ago and it already fired down here. Hello from the past. So the other thing Alvaro was talking about was splay so if you don't want, if you're managing like a fleet of hosts and you don't want every single timer to kind of start at the same time because that would overload whatever service you're trying to make them call or something like that, system D provides a way to automatically randomize when the timer execution actually fires. So to do that you can use a randomized delay seconds. I also like to combine this with fixed random delay because if you have fixed random delay equals no, every time you actually reload the unit, system D will change the amount of randomization that goes into when the timer executes. Yeah, it'll make it random every time so it's not gonna have like a consistent or random delay. So let's look at this unit as an example and its contents. Can someone tell me what it does? Brave Soul, any Brave Souls? Sir over here with two hands. And say that it will run every 30 seconds but with a random give or take of 10 seconds but probably the same random give or take of 10 seconds every time. Yeah, that sounds good enough to me. Congratulations, you win portals. You're also, I actually don't know, I'm actually bad at reading crawl tabs. I believe this is 30 minutes, right? Yes, it is, yes. So this will run the timer every 30 minutes. Sorry, not totally correct, you didn't say seconds. No, these are seconds, right? These are seconds. Okay, nevermind, I'm wrong. It'll run the timer every 30 seconds adding up to 10 seconds of random delay. So extra stuff we won't cover.  Ooh, actually one thing about time zones. This is a big gotcha. When you use on calendar, it uses your local time zone. I know, it's crazy, right? So like when daylight saving comes around and your timer doesn't fire for like the next hour, that's probably why. So on calendar uses the calendar. Yes, yes, I mean it sounds obvious like on calendar but that's a huge difference from the crontad behavior. So you can actually stick a time zone on the on calendar if you like really don't want to or you can use them on atomic timers which are based on your system. All right, things in UTC? UTC zero. Okay, so, yes, monotonic timer. Time is an illusion, it doesn't matter. All right, so we're not gonna go over every single unit but the other interesting one you probably want to know about are path units. So what path units allow you to do is kind of watch a file or a certain path and then run like a service or something when that file changes or it's modified or it's created. So let's go ahead and create this SH config watcher path and I'm gonna copy the contents from our GitHub by clicking the button. So let's look at what we have here on this path unit. Similar to service and timer, it has kind of a path type or header, I don't know what you call this. It has a path change property. So what we're gonna do is watch when ETC SH-SHD config, the contents are changed and saved to disk. And of course we need the corresponding service unit along with it, so we're gonna go ahead and do that. So just copy it from GitHub. So what the service is gonna do is when the path unit fires, we are going to systemctl reload shd.service. I got a good question during the break about systemctl reload. So certain service units define a reload behavior. SHD is one of them.  You do something to reload the config or whatever. So let's go ahead and save that. And now we can actually, we'll go ahead and enable the path unit and then start it. Now we can check the status of SHD, just to see where we're at. And you can see right now that SHD is just running normally. But if we, let's say we just modify the ETC SSH. SHD config. This is the file that we're watching on the E-PAT. Let's say that we uncomment these lines here to change the log level and then save it. If we look at the status of SHD again, you'll see that the process actually changed to executing its reload behavior, which is what we actually defined in the steps above to create the path unit and the service unit. And also the other interesting thing is that SystemD tells you when it reloads, so you can see down here that SystemD had reloaded SHD. Any questions about this? And then of course you can use SystemD Run to also create an ephemeral path unit like you did with timers and services. So let's just go ahead and copy this one. Instead of path changed, we're using path exist here. What we're gonna do is when we create slash temp slash ping, we will touch slash temp slash pong. Let's go ahead and run this. And we can check that both ping and pong do not exist right now. So if we go ahead and touch ping, we will see that the unit ran and it created pong. Okay, so I have something insane with the path exist behavior. So you would think that when you use path exist, it only runs the service once, like once it creates the file. But if you read the documentation for path exist really closely, what it does is while the file exists, SystemD will keep trying to activate the unit. So if we look at the service here, the service that it created as part of the run, we can see that it actually hit the start limit because while that file was existing, SystemD kept trying to start the unit. At some point it hits the start timeout limit and then it dies and it stops trying. So yeah, that's something you should notice, I guess. Yeah, sorry, it is very useful. You have that gotcha. The other one is that this is using the, what's the name of the thing where it's, I notify, yes. So as you know, I notify it's a best case effort. So you may lose some events of it. So I would not extremely rely on it, but it is very useful for things like let your users add a file into the directory and then you can grab that file and then shove it into another configuration or something like that. We did this for SSH once in the company where Chef would write things into directory and this will basically create an SSH config and reload SSH when it starts. You can see that that is very dangerous and cloney, so don't do it, but yeah, that's what we did. So that's the end of the other unit types that we're gonna go over today. Are there any questions about that? Nope, all right. I'm gonna break or should we keep going? Okay, all right, so the next module we're gonna go over is dependencies and for some reason Alpharil's really excited about this. So dependencies are, I mean, conceptually they seem kind of like easy to understand but in practice they can get pretty complicated. So the, if your members is via NIT scripts, you run the scripts sequentially based on how you number them and there's not really a way to say that, hey, this script depends on this other script except to order them in a certain way based on the file name. With this system D aims to solve that by creating a way to define your dependencies and when system D will try to like graph all the dependencies together, it tries to parallelize the execution based on how the dependencies are kind of linked together and that's how you can speed up, start up and shut down. So instead of running everything sequentially, it aims to parallelize as much as possible. So what does this look like in practice? So let's start with ordering. System D provides two ways for you to order your units with before and after. So for example, we're gonna be talking about the dinosaur and human service. In the dinosaur service, we are actually going to write before human.service so that when these two units run together, we actually have dinosaur.service running before human.service. So we actually provide these units for you already so I don't have to struggle to copy and paste the contents into the terminal anymore. So we'll be able to just run this and see what happens. The thing, the other notable thing here is that on both of these units, we use remain after exit. I don't think we talked about that yet. What this does is it keeps the state around after the service finishes running so that you can examine it once system D has in memory. This is pretty useful if the service successfully exits and you wanna see what happened. So let's go ahead and copy the start command and to show that dinosaur ran before human, we're gonna check the timestamps for both of these units. So we can do system CTL show dash P. We use the active enter timestamp monotonic property and we will first look at the dinosaur.service timestamp and then we'll look at the one for human service. And you will see if you look really closely at the numbers that the dinosaur service had activated before the human service. So yeah, super simple so far, right? Before and after? Okay, now we get into kind of the bread and butter of like the actual dependencies. So 90% of the time I'd say most people only wanna use requires and wants on their units. System D actually provides many more ways to define dependencies, but those really get into like the nitty gritty of how you wanna start, stop and reload on certain units. But most of the time you just wanna say this unit wants this other unit to start or this unit requires this unit to start. So we're gonna test this out with two services, life.service and water.service. On life.service I write that it requires water.service and we also want it to start after water.service. And water.service we're gonna run bin false so what this does is it causes water.service to fail when it runs. And you'll see why I do this once I start life.service. So because I use requires we would kind of expect that life.service will fail because water.service is going to fail, right? So let's start. This is another gotcha here because when you check the status of life.service and also the status of water.service you'll see that life.service actually ran successfully but water.service failed like I said it would. Why is that? Well it's because requires only refers to the act of starting the unit. The default type for every service unit is simple which means that as soon as system defaults the unit is considered started. Which I think can be confusing because people have very specific ideas about when their process is actually started, right? You would think that when we exact and we reach the certain point in our process this is when we want to start. So to do that we can actually use type equals notify. So if we system CTL edit water.service and we change it to type equals notify because in our command we don't tell system D, sorry taking a step back. What type equals notify does is it expects your service somewhere in your code to tell system D when the unit is started and ready to take, actually active. You do that by sending a command, sending a D bus message ready equals one to system D but because we're not doing that here we will actually fail to start the unit. We have a tutorial on that tomorrow. Chambers play. Now if we actually try to start life.service it fails because the water service failed and it had a required dependency and you can see that system D helpfully tells you that dependency failed for life.service. So in the real world this is useful like when you want to start your web server after you start your databases like this is how you can do that. In general this is how system D starts in general. System D says like start everything on the service that it's designed to start and then it use dependencies to map things out. Now we have a quick exercise for you to do.  Just take like two minutes to do that and let me know what you come up with. Okay so who has managed to send the ready message and like gotten the dependencies all fixed up? Does anyone want more time to try and do that or do you want me to do it? Ambivalence I will go ahead and show you how I would do it. So I'm gonna override the previous exact start by just changing by adding a blank exact start here and then I will use system D notify. If you looked at the man page you'll see that you can send a dash dash ready. And that's all you really need to do. So if we start life.service again you'll see that it started successfully even with the type equals notify and also water. If you look down and we deactivate it successfully. And to do that in your code you would use like the SD notify API and things. Oh there's a question. Yes. The question was can you show the solution and the answer was yes. Yeah here's the solution once again. And again we have a whole section of notify in general. There's a lot of cool things you can do with it. So stick around for tomorrow. We talked about requires. Now we're gonna talk about wants. So wants is basically the weaker version of requires because what wants says is that we'll make a best effort attempt to start the service but if we don't do it like no big deal we'll just keep moving along with the original service. So we provided two service units here for you. Dog.service and bone.service. Let's see. Bone.service has the same thing we had in the previous section where we use type equals notify and we run bin false so that it purposely fails to start the unit. And in this case when we actually start dog.service it will run successfully because we don't have a hard requirement on bone.service. So let's just go ahead and do that. And if you check the status of dog.service you'll see that it ran successfully and bone.service had actually failed. So yeah. Any questions about the ordering and basic dependencies? Yes, question. Oh, I guess the question was if you define the unit such that you create like a cycle or a deadlock is there a way to detect that with system D? Yeah, yeah. So like two units like depend on each other and if one does like, yeah. Yes, you, so yeah, I mean you can create cycles on purpose and system D is able to detect and warn you about it. I don't know if we'll get to the verify section of the workshop today, but you can use system we analyze verify to detect cycles in your units. And you'll see that the crazy thing is when system D detects cycles on boot or like at a target it'll actually delete the start of certain units in order to get past the cycle. Any other questions? Yes. Oh, does the unit know if it was started by a once or requires? The answer is yes. There's a lot of conditions in which you can basically say because like one of the things that you can start unit just because it's started by requires or wanted but if you stop the main unit you will not stop the previous one, right? But you can also set system D to say like if a user doesn't start this like stop it when you don't need it. You can also say no one can start this unit except another unit by once or require. So you can also like further. So the answer to the question is yes. And you can take advantage of the situation. I'm gonna talk about the install section of the unit. So you saw in the previous examples that we added install so that we can actually enable a unit and most of the time we do that because we want a unit to start at boot. So typically we add this statement here install wanted by multi user target. For most systems multi user target is the default target that kind of signifies like the end of the user space. Like we booted into the system and this is it. There are actually multiple other targets you can do but let's start by modifying. Yeah there are actually multiple other stages of targets that you can do and let's actually do a quick experiment here where we add the multi user target to the dinosaur and human services that we did before. We're gonna go ahead and copy dinosaur service into slash ETC system system and name it a dot service instead. And then for human service we're gonna call it the service. Okay so here's the disclaimer. This is the part that we haven't tried with lean out.  That's how it works and we don't end this story also earlier. I did actually modify and add the install at this point and the point of rebooting here is that we're gonna show that both of the units did not start at boot. This fast in my vagrant machine I don't know how long it'll take your lean out machine to come back up. I hope you try a restart and it takes like a minute or two so. Did anyone do a reboot on their lean out instance and come back successfully? So if we do system CTL on an beta service you'll see that they did not start as expected because there's no install on those statements. So we can go ahead and add the install now. Saving and we also do the same for, so what I did just now is add the install I want to buy multi-user target part that I had above in the GitHub to both of those units and net, oh, don't forget to system CTL enable which creates the sim links on multi-user target because if you don't do that it is not gonna start at boot and if we actually reboot this time you will see that both of the services are gonna start at boot. Any questions about that or anyone need help? The question was what are the constraints between? Oh, what are the differences between adding the install statement and running system CTL enable? So you can only enable units that have the install section. You'll see that if you try to enable a unit that doesn't it'll give you a long log line or warning about it. Basically what the enable does is set up the sim links so that when system D runs the target it knows that these are all the different dependencies or services and stuff that it has to run. So getting to the next part of this we're gonna start talking about synchronization.  So targets is another unit type that doesn't allow you to specify anything about it except for like the default unit properties and they exist to synchronize all the different services and stuff at different stages of boot. You can think about them as like this is fee and it run levels. If you actually look at the man page for boot up so just go to like man boot up and scroll down a bit. You'll see that there's a really interesting chart about how all the different targets get executed as part of a host startup or run. And you'll see that at the very end of it the user space boot targets can be either one of rescue target, multi user target, graphical target or emergency target. Graphical is like when you have GNOME, KDE or some kind of graphical desktop environment. If you're on a server you don't really have that so multi user target is where you end up. And you can actually add your own target if you want. So to see what the default target is you just go run system CTL get default. And you'll see that the default target is multi user target. The way the default target is defined is with sim links as with a lot of the other kind of dependency things. So if you look at the sim link for default.target you'll see that it points to the multi user target unit file. So the next part of the workshop I just show how we're gonna create a custom boot target but in the interest of time I'm gonna skip that part now and just move directly into the showing you some examples of like how you can analyze boot with system analyze critical chain and things like that. So if you run system analyze critical chain what it does is gives you kind of a rough very rough time stamp of when each target ran when you booted your host and the order that they actually ran. So in this case we still had multi user target and these are all the things that had run before it along with the time stamp. And if you like pictures and pre things you can actually use system analyze dot pass it a unit name and it'll generate this crazy graph of all the dependencies on your unit. System analyze dot, I don't know what's our favorite service. So you'll see that this spits out the dot format for the dependencies of system d umdi but this is not really useful by itself we are visual creatures so they say that we can actually use this to turn it into an SVG. It's gonna run that again. And if we have an image viewer we can actually look at this crazy graph. Here's the Linux desktop everybody. So there are actually not that many dependencies associated with system d umdi. You can imagine if you do a more complicated service this kind of chart gets crazy and hard to look at but just even like this it's a lot of the default dependencies and not even the ones we specified specifically. And then the other tool is like system analyze plot which I am not gonna do but you can run it on your own time. It will generate an SVG of the entire boot sequence and it's like a multi-page visual of like how the time that all the units started. Any other questions? Okay, we are gonna take a five minute breaks, bathroom breaks, relax breaks. We are present time we are gonna finish here at one because of respect of people who work here and stuff. So keep it five minutes and then we'll get back. Okay folks, don't forget we still have some port of space trees that you can go and take. Come and take sorry, what is your problem? Okay, I assume we are on the right time. So we are gonna finish this strong. We have about 30 minutes to finish. We're gonna do now the beautiful world of journal CTL, right? So as a little bit of introduction besides of all the controversial part of system D system D also decided to replace like Syslog and other the journal CTL. The value study offers to people who manage system is really incredible, right? So you can still like not like it, that's up to you but it is really good to operate on it. So we're gonna do a tutorial on it where we're gonna give you very basic dive into journal CTL. We have been using it a little bit and then in tomorrows which is the advance we're not gonna teach you advance part of journal CTL but its commands are gonna just show up on the tutorial. So it is nice. So we're not further ado. Anita will start this and then I will finish it. Like Alvarez said, system D comes with a daemon for doing logging and it's called system D journal D. The way you configure the system D journal D daemon is with the journal D.com. It's stored under the usual places slash Etsy slash and D and you can look at some of the commonly used properties along with their defaults in the example that we have in the GitHub here. We're not gonna go into configuring any of them today but I'll point out some of the popular ones such as storage which you can set to volatile which will only store logs in memory persistent which will store logs on disk auto which will try and see which directory exists in order to decide if it wants to be volatile or persistent and then none where like we don't store any logs at all we just forward them to wherever you tell them to forward to. There's also a global rate limit that you can set with the rate limit properties. You can also set per service log rate limits so that if a certain service is too noisy you can tell it to shut up faster. And then you can also set the properties on how big your journal should be on the host. Skipping these, there's also the forwarding behavior like I mentioned, you can forward to syslog, K message, console or to wall and they're all pretty self-explanatory. And you can also set the max log level of the messages that go to the journal using the traditional syslog levels or with the integers. So on a per unit level you pretty much only have like six sysh properties to configure how your unit gets their logs into journal. By default all of standard out goes to the journal. You may not necessarily see all the output from standard out in the journal because of how the log levels are set but that's how it is. And standard arrow by default just inherits based on the behavior of standard out. And like I said, you can also set individual log levels in the service unit and also individual per unit rate limits for your logs. So enough about that, let's look at some logs.  And yeah, just a way for you to page through the logs and things like that. Pretty straightforward. I usually like to do journal CTL R. What it does is it reverses the log so that it shows the most recent log on top because I don't really care about the logs from like two days ago necessarily. I want the logs from right now. So that's what R does. And you can also use dash dash since, give it some relative time stamps in order to be more specific about which logs you wanna look at. So if we do dash dash since, these are logs since one hour ago. In 101 we actually did the last 30 logs. So I'm not gonna run that again here. And you can also use journal CTL to look at journal file that you kind of imported or moved over from a different host. So we provided a journal file for you to look at just as an example. I'm gonna make sure it still exists. You can pass dash dash file and you can use journal CTL to look at basically arbitrary journal files. It's pretty useful if you need to debug something. A user can just sign you your journal file to look at. Okay, okay, so now we can see a little bit how can you as a developer write to the journal, right? So we already saw that your unit by default is under SDD in, sorry, so there are SDD out and SDD error to the journal. We provide a small unit over here and you can go and instead of start first, let's see what it has, right? So as you can see, it's just saying standard journal, hello, and now we can start. And that will, if I do status, you will probably see that, yes, it is right there, perfect. And now if I were to do journal CTL that you, you will see that it kind of shows everything that goes into that unit. You can also see that it shows other stuff that are like from different PODs, but these are all in the context of the unit. Give me a second. The other thing that we didn't say, but we kind of explicitly have been saying is that when you decided to add a unit in your commands like here, you can do the unit with service or without the word service. So without the dot service, it just appended at the end, right, so it's figured out, right? The other thing that we didn't tell is that we have set up a tab competition. So if I do journal and then press tab, it autofill it, so you don't actually need to type the entire thing. This is, this comes part of system D. When you install system D addons, extension, what's the name of the package that you install to get tab competition? Bash complete, oh, I think it's come with system D. And if you install this competition, it comes, yes. It just comes with system D. Okay, and that's moving on. Michael, so that is one of the ways that you can print to the journal, right? So I also provide, sorry, we also provide here as nice script that you can look at it and this is how you can write to the journal with, so if you look at that file over there, you can see that is importing system D journal. We're gonna talk more about system D next, like tomorrow, but that's a Python interface to interact with system D. And basically we're gonna say journal and send the variable and we're gonna send this message. We're gonna set the priority, the message and the syslog identifier for that one. So if I go here and just execute this script, like remember, I'm not even like, technically you're always in a unit by, you can do that by looking at just, like doing system D status on the double pound and that's your own PWAV. You see that I'm literally on a unit, it's just a scope. But yeah, so if I go and execute this nice command and then do grep for hello, you're gonna see that I had a message there from super unique ID and then maybe that's the syslog identifier and then hello from system D journal printing, right? So if you kind of see, that's the identifier that I have. So that's how you can like write to the journal from anywhere that you want. We already saw that we can do this by unit. What we didn't saw is that you can ask for the output to be verbose. So this is also another interesting thing that the journal has, that you don't see it everywhere else.  There is a bunch of metadata under the hood, right? So when I do verbose, you can actually see a bunch of more variables being set. So you can actually, like since you can read this, you can do stuff based on whatever the message was or the login, you see like there's a syslog identifier there, but it also kind of tells you like what was the command, the host name, the message it says, it kind of gives you like the PYD and the command line that was executed and you have the sysgroup and the unit from which this came, right? So there is a lot of information. So you can look, we already saw that you can look for a unit but you can also probably search by sysgroup or any of these filters, I can go. So yes, that's interesting. Oh yeah, we can also do this, journal CTL feed one and this shows, we are searching, like we saw for filters. So we're using everything that comes from P1. As we know, that's the unit system, system D. You can also do this. So look at all the messages that come from a unit but from the PYD, right? So if I delete the PYD, sorry, this is going to be if I, if you see like there's some methods that come from the thing that was exec but system D itself add logs to that, saying like the things that it was doing. So if I just want to see that because sometimes it is useful, you can do PYD one. This doesn't work, right? It doesn't work, right? All right, cool. Yeah, we have some exercises here that you probably can go and look at them. The first one is view the metadata from that file again and you can see what files there was. We're going to leave like one minute or two for people to do this and then we're going to do it ourselves.  Okay, so as probably you already figured this out, you just need to mix everything and then you can just do that and then it will show you everything that was running. It's ask you, okay, so let's take this one. We can basically take that one and then say unit and perfect, the PYD was 333. Since you make this part, am I right? Yes, I got Anita's approval. I got, so any questions so far? No, okay, great. So let's do the last things, right? General CTL, if you do this Mac, you get the message from the Deepak kernel.  General CTL list boots shows you the boots of the machine and finally, if this one is not a good example because it only has one boot but if you do with that three, it will show you the logs for the current boot. That's the journal introduction. Go ahead, you can mix and match. Like for instance, one thing that you can do is that like this one, you see has like file solution but you can also add another unit, right? I don't know if systemd, umd, yeah. So like, and it's fine, I'm just gonna show you the journal, probably umd wasn't running on this one. But example, let's do it here though. There's journal, right? Just press U, you said, okay. Yeah, all right, believe me, it's somewhere there. That's the, yeah, that's the one. Okay, go ahead. So the question was, sometimes you do systemd cut and it looks as echo. Yeah. So there is a bunch of things in play there when you're logging to syslog. The first thing is that systemd cut and some things try to log as the main piece of the process because like it's assuming that systemd cut is not the one that is doing the log. So it provides their information to say like, oh, systemd cut is the one, by the way, this is what we're talking. So yeah, so systemd is, so it's, this is gonna go and show and try to put stuff into the journal, right? But it makes no interest to say like, oh, the systemd cut, the one that is showing. So systemd cut tries to operate as the parent piece. If you are running as a user, like niche, by the way, niche, the, whatever you start doesn't have permission to execute as another thing. But if you're running as root, you can.  This is very common when you do systemd notify, which is the thing that we're gonna look tomorrow. But yeah, so there's a lot in place. The other thing that has to be in place is that by the time journald sees the message, the PYD that sends the message has to exist. So there's a race condition there where you cannot have attribution on the logs. It is, it's a problem. Yeah, yeah. So yeah, so what Nish was saying is that he basically tried to do a denial of service on his journal CTL and some of those messages got misattributed. Yes, this is something that happened. And I think we are done for the day. So who here has learned something new today? That is great. We're very grateful. We really appreciate that you guys came here. Thank you very much. And have a great rest of your day for the conference and see you all tomorrow morning at same time, same place. Okay, so we have 15 minutes and we have one more example. Do you guys want to see it or it's time to go home? Okay, let's try. All right, let's, all right. Disclaimer, we did this for Fedora 37, but let's see if it works. Okay, so the example is the following. Go into, so this is the one that we're gonna do. It's the workshop, workshop, analyze system, the analyze engine X. So we're gonna do a thing that is called system, the analyze security. If you type this into your terminal, what this is doing, it's giving you an opinionated, an opinionated by system D of how secure your services are, right? So the definition that system D has for secure has a lot to do on, let me take a step back. So in the old days, you start your service like you start an application and this application or this service that is running started with the same privilege as you and here it's everything that you could do, right? It has the same view of the file system. So if you can write into root, the service can write into root and all of those things, right? Today, this is not necessary, right? So if you're having, if you have an application that all it needs to do is check if you have unfinished downloads. There's no reason why it have to be able to write into Barlin, right? So you can actually hide a lot of the system, right? Maybe your application doesn't need to even see home directories because like, why would you get access to your SSH keys to a random service? So you can actually hide them. So all of the things that system D provides for you, if you don't enable them, they called your system unsecure, right? Again, it's opinionated. Let's pass that, let's get passed that. But you see they have some services here and we have a lot of them and they have some data in them. So we're gonna try to do one with NGINX, right? So let's do system D analyze security NGINX and it's gonna show you all the things that it thinks are broken for NGINX, right? So it shows you all the properties that you can set that would change a system D, the NGINX behavior. And it has something that, oh, this is great. It has private TMP, right? So that means that TMP itself, it's mounted as a RAMFS. So NGINX doesn't have access to your host system D. TMP just has like a shim version of it. So if one applications store stuff there, NGINX cannot see it. That's kind of like the whole idea, right? So before we start, let's try to start system CTL NGINX and you're gonna see that it fails, right? And it kind of tells you, hey, execute this command to see why it failed. And it says, it says very not useful information. It says that it failed it to start because it failed it to open bar log NGINX error log, right? So what we're gonna do is that we're gonna copy the NGINX config, the previous NGINX config, and then we are going to edit that config to do in line seven, line seven, we're gonna change that error log. Sorry, line seven, we're gonna change this to be that. Basically, we're saying instead of like storing logs into a directory because we actually want to like avoid this NGINX to write into stuff, we're gonna say just send it to syslog server, which is just journal D. And then we're gonna do the same thing for the access log that is on line 22. We are also, sorry, we're also going to edit this NGINX unit, and we're gonna change the log directory to, this is another unit, this is another setting that is just gonna set a unit, a directory for the logs. Okay, and now we can start NGINX, finally start, and then we can do a curl to port 8080. And if it returns okay, it's like, we got NGINX to work, that was not part of the tutorial, that was just like cloudiness that it comes with. Okay, cool. So now let's try to improve a little bit of NGINX. So we're gonna edit the NGINX, and now we are gonna add these following settings. These settings are actually kind of cool, what they make, and this should be on for most things. First of all, it makes private devices, that means that everything that is in your bar log, that has to do with physical hardware, sorry, bar log, depth, or depth, that has to do with physical hardware, it's not allowed for the unit to see it. Like for instance, there's no reason why NGINX has to write directly into your CD room, or your hard drive. So we're just gonna say you're not even gonna see NGINX SD1 or NGINX SD2, you don't need that. Private TMP, I already announced this, it basically, for TMP, it mounts a TempFest, that nothing has access to it, and only NGINX has access to it, so there is no cross-contamination. Protect Home makes Home, I think read-only, I don't remember what through this, but there's options where you can make it hide home, the other one is just make it read-only so that nobody can write it. And this one are other options that has to do with kernel, like NGINX is not gonna load a kernel model, so why even, I'll give it the chance that it could do it? So all of this, right? So we save it, we restart it, we check that we can still query, and yeah, we can still query, so that exists. So let's see if we can analyze NGINX again, and we go to the end, and we now see that the overall exposure went up, right? That's not the exposure, but the number went up. Now we can do a couple of the fun ones, right? So we can see if there is a socket, so there is no NGINX socket file, and I really just realized that we haven't actually looked at what socket files are, but we can go and create a socket file, I have to say force. Yes, so with these options, we will create a socket file, and then we can add this content. What this is gonna do, it is very simple. So NGINX start as root, because it needs to open port 80. So instead of doing that, what we're gonna do is that, sorry, NGINX start as root open port 80, and then download itself to whatever user you tell it to run. But we don't need to do that. What we can do is just let systemD start listening to port 80, and then with NGINX start, systemD will just handle that file descriptor to NGINX, so that way NGINX never even have to be root, right? So this is what we're doing right now. We're creating an NGINX socket, we're setting listening string to port 80, we are exiting BIM, and then we are going to edit the NGINX, and we are gonna add the following options. For the unit, we're gonna do what Anita just told us to do, which is set up after on requires, and then we are gonna add the following things to the service, we're gonna say the type is simple, we're gonna say that please don't rely on PWAV files, we're gonna set an environmental variable, this is the one that tells NGINX to read the sockets from file descriptor three and four, and we're gonna say private network, so that NGINX cannot access other things other than this port, right? So NGINX itself cannot open port, so if somebody hacks NGINX and can execute stuff, they cannot start listening to two other that is not 80. So that looks great. We are gonna stop NGINX socket and service just in case, and then remember when Anita was showing you that you can start the timer and then the timer will start the unit or the path, this is the same, right? So we're gonna start the socket, not the service, you can start the service, but we're gonna start the socket. Now, if I do status on the socket, it says, oh great, it is running, right? It's listening, all the information is nice. What if I say the service? The service right now you say that it's loaded, but it's not running. It's gonna run the first time that somebody hits port 80, it's the same as the timer, right? So when timer gets to a time, it execute the service, the socket, when somebody hits the socket, it's gonna start the service. So if I had eyesight, I could have done it, then I could paste this, and then yes. And then as you can see, it took a little bit more the first time, and this is of course because it started the service. You don't have to do it like this, but it's kind of useful because up until you actually require the service, you're not wasting resources on it. Question. Yes, so the question was can system be, or is there a way of preload the service before somebody asks it? And the answer is yes. You just start the service on the socket at the same time. So where's the stop? Yeah, so you can actually start. Oh, yes, so you can start it, and then the status of the service is up. So now my, whatever I do by curl should also be instantaneous, right? But it's still going through the socket, right? And if I understood Benita's tutorial correctly, I should be able to just start the service. And yeah, and by just the sheer magic of dependencies to start the service, system is so that you actually want the socket before starting the service. The beauty of the socket, it is like if you're not latency bound, you are saving a lot of resources by not having something running. Okay, so this was it. Okay, and I want you to see that we're doing all of this because we want to, there you go. We want to minimize that number here. We want, this is the one that we want to improve, right? We're already improving a little bit. So let's do, let's add more stuff. So now that we actually don't need to be root, we can just, now that we don't need to be root, we can just set these variables here, and what they all do is very simple. We're saying, yeah, just start as nginx and group nginx, don't even think about being root. Like we, the only capability that you have is that you can bind to the network. And this here over here says, like systemd will create some directories for you that you can use that you don't need to create. Like bar run nginx, like nginx doesn't need to create this directory. It's already there by systemd. The same with the state, barlib, barcache, and etc, right, for the configuration. So now nginx doesn't need to be able to read or write these, sorry, to write these directories. So to create, okay, we save it. Now we can go and remove the user nginx from the, here there is a line that says that the user's nginx, we can go and remove that line. Yes, that's great. And we can also remove the line pyd, this one over there. We can go and delete the locks directories. We no longer need it. We can stop. We can start the socket. And then we can do analyze. And now we have a happy face because systemd really looks images, right? So the security just went through to forward eight. Yes, that's still the case. And now if you really, really, really want to overkill it, we have like more options that you can go down. Again, we're two minutes. And I think that takes us to 4.0, right? So this is 4.8. If we do this, it will take us for 4.0, but how much secure you wanted to have it, that's up to you. But the point is that you get all this feedback loop of you add stuff, what it's doing. Keep in mind that this is one definition of security. It doesn't really have to be the one. But as long as you can still curl your service, and it, okay, that wasn't working. Let's assume it was. All right, but as soon as you can get your service to run, that's what you want. And on time, so thank you, Aaron. Howdy, can everyone hear me good? Excellent. Hello, hello, hello. Okay, give it a second. Hello, hello. Ooh. Yeah, we can kind of stash in. Oh, wouldn't you care about something here? No, I mean, we've got, yeah, we've got sessions. Yeah, yeah. But that's when you've got sessions that need to stick inside of there. Yeah, we'll all be able to do that. Yeah, we'll all be able to do that. Yeah, we'll be able to do that. Send out the viewers if they want to. Hi, everyone. Thank you all for coming, for coming to scale. I always enjoy being at scale and for coming to this. This is a new kind of format we've been experimenting with to do kind of an overview of CentOS and then a workshop so you can get some real hands-on experience with packaging and doing stuff in CentOS. So we've been wanting to do this. We've been wanting to do a larger presence at scale for a while. So very excited to be here and doing this. So what is CentOS? Now I'll have more slides on what is CentOS. But we have a whole ecosystem. What you might refer to as just CentOS, is CentOS Stream, that's the operating system. That we'll talk about more. And that's kind of in the center of everything. But then we have all of these special interest groups around it. And you can see that I'm gonna get a graphic designer to actually make this look pretty at some point, but I made this so it looks terrible. You can see all of our, well, that's not even all of them. There are more special interest groups than that, but that's some of the more active ones that do various things, build different package sets, do alternative packages sets, other images and stuff around CentOS. Oh hi, my name is Sean. I am the CentOS community architect. And here I'll let Carl introduce himself. How do you all? My name is Carl George. And I'm participating in the CentOS project, but also focus a lot on the Apple project, which we'll talk about on another slide. I think the logo's up there right now. And also I'm not on camera frame, like I was just complaining about.  It's not part of the CentOS project, technically. It's part of the Fedora project, but very interrelated. And I'll talk about that more on the slide where that's appropriate. But yes, hi, I'm Carl. Yeah, so I will also get a graphic designer to make it. I wanted Apple on there because they're within the ecosystem, but they're not technically a CentOS sig. So let's talk about CentOS Stream. CentOS Stream has a five year lifecycle, a five year and some months lifecycle. There's a new major version every three years. And it is where you get enterprise Linux from. It is where REL is derived from and any REL derivative systems ultimately come from CentOS Stream. It branches, I'm just speaking and not following the slides. It branches off of Fedora. So every three years we take a Fedora release and we make a branch point from that. And this year actually will be one of those branch years. So Fedora 40 will be coming out in a month or so. Like really soon, right? So Fedora 40. Oh, okay. I will hold that thought. Okay, so it's a long lifecycle. We have a five something year lifecycle for CentOS. So there's about a two year overlap between when one version of CentOS, the new one comes out and the old one is still there. It is not a rolling release, even though that word has floated around even on some of our own blogs early on. I think people use the word rolling release to try to indicate it doesn't have minor versions. There's no CentOS 9.1, 9.2. It just has CentOS 9. But that's not what people think of when you say rolling release. When you say rolling release, people think of something like Fedora raw hide or opens is a tumbleweed, right? Where there is no major release. It's just constantly getting updates, right? And that's not what happens in CentOS. In CentOS we have a major release and it just gets updates within that major release which is effectively the same kind of release cadence that you get on Fedora or most other distributions. They have a major release and then they get stable updates onto it. And the updates that come onto CentOS Stream are stable updates. These are generally updates that you've already seen in Fedora, either the update itself or it might be a back port of something that's already in a more current version, right? We maintain sometimes older versions of packages within CentOS Stream so we have to do back ports. But the features and the fixes that you see coming in are things that have already, that you've usually already seen in Fedora and they've passed a bunch of gating checks and everything so they're not experimental new features. They are stable updates. I'll interject there. The way I describe it, I might have put it on the slide before maybe. Yes, that last bullet point. I would call them rail style updates where if you're familiar with how that works we do a thing called in rail called back porting where the version of the software stays the same but a bug fix or a security fix or something will get added as a patch to that older version. So you're like the software might be version 1.0 but then there's a security fix that came out in 1.2 but there's also a breaking change in 1.2. We want the security fix without the breaking change so we take just the changes we need for the security fix and back port them basically making a soft fork of the software. If the software ever got rebased like it does in Fedora raw hide you could drop those patches but it typically, I don't wanna say typically there are some packages that get rebased in CentOS Stream. I know that the Rust stack for example moves very fast and that gets rebased in every real minor version so that'll get rebased in CentOS Stream first a few months ahead of time but for most packages there's a strong preference to back porting things and keeping the same version for stability versus rebasing to a new version. And the kernel is like we stay on I think we're on the 514 kernel. So we have 514 but it's like revision number 200 or something like that's how many back ports get put onto it. Yeah, let's do, gonna run the mic to them. Test, test, that's sensitive. Yeah, my question was you don't update like the package version, right? But the only way you'd know the back port was done is to either look at the change log or to look at like the sources for the patch in like from the spec file list, right? Like looking at the spec file or? Yeah, I can answer a little bit of that. Yes, that's sensitive, I'm just really loud. Definitely the change log. Also, if you stay for the whole session we're doing an RPM packaging workshop where I'm gonna get into that in depth detail. It's the field called the release. If you've ever seen the string for the package it's like 1.0-5 that what I usually tell people is the first part before the hyphen is the upstream version and the other number is the package version. So like the revision of the package. So the first one would be dash one and then you need to add a patch and then it's dash two and then you realize you forgot to ship a file that's supposed to be shipped with the software then you do that and dash three but it's all still version 1.0 of the upstream software is where you're starting from. I think it's a little bit misleading when like you mentioned in the kernel where you're on like dash 307 and because there's so many patches added the upstream version is less and less relevant when you patch to that degree but in a long life cycle distribution like CentOS stream or rel that it's unavoidable because you need to keep a lot of those versions the same for a long period of time. That covered? Definitely jump in with questions and I'll come out to you with the mic. Is the order Fedora first then CentOS then Red Hat? Next two slides. Do we have that on the, that one shows. Okay, a couple slides ahead. Put a pin in that and yes. I have a sticker actually I have here. Talk about it. I'll go ahead and talk about since we had a question. We don't, it doesn't really matter what order we talk about these in. So this right here, can y'all see that laser pointer on the screen at all? Good. So historic, a lot of people come up and ask I thought CentOS came after rel and I thought it was a rebuild of rel and traditionally that was true. In the past we had the Fedora project a version of Fedora would get taken basically taken internally private inside Red Hat and worked on for a year plus and then all of a sudden kind of throw it over the wall open source that many engineers aren't a fan of because it's, you're technically going from an open source project to close development and then dump it over the wall open source. Like people that care about open source know that that's flawed and don't like that but that's how rel development worked in the past. It was all private development and then thrown over the wall. Ta-da, it's open source but you can't really do anything with it other than use it. The CentOS project will come along and use those sources to create a what we would call a clone or rebuild of rel and I'm gonna go back a slide to talk about that one but that's not how it works anymore. The new model that is a lot more sustainable and more interesting for people that care about open source is that we have Fedora and then it branches into CentOS. There's a period where it's like the early development and you probably don't wanna start using it day to day yet and then at some point we actually announced it as here announcing CentOS string 10 and then about six months after that that's when you see rel 10 which is, yeah, so here's how it worked in nine a little bit more granular view. Fedora Rawhide is the true rolling release in this ecosystem and the dashed lines are gonna be like development so you'll have, not right now but Fedora 34 branched off from Rawhide got a little bit more polish and then released as Fedora 34 maintained for a set amount of time. At the around the same time we branched off CentOS 9 development worked on that for a little while getting more polish and then at a certain point rail nine beta branched off of that and then we did the CentOS stream nine announcement and then a little while after that rail 9.0, 9.1 and so on branched off from CentOS stream nine development or actual CentOS stream nine. Things change a lot between here and here we're getting things stable for the whole life cycle which the rail engineers controlling those packages they have to think about it not just what we're putting in the next version of CentOS but what is the business gonna have to support for 12, 14 years depending on the extended stuff that they sell on that side of the house. And that's what it's gonna look like for 10 and this is, we are actually right about here right now we're in the last bit of the Fedora 40 development thing we're about to break off the sync we've already got the first initial composes to make sure everything works for CentOS stream 10 but they're nowhere near the point where you should start using them. Once, once Fedora 40 is released we're gonna interrupt that sync we're just using that to bootstrap things and then independent development will happen for 10 things like deciding exactly what version of the kernel they're gonna use and exactly what version of OpenSSL is gonna be used things like that that all is gonna happen right around in here but once we actually announce it as it's here it's ready, you know no one's ever gonna say it's ready but we'll announce it and that'll be a good time to get on board and start using it and that's around the time of the five and a half years we mentioned. So going back to where we were before the great questions, keep them coming. Before I go on, was there any more questions about that kind of life cycle interaction in the branching? I know it's a lot of information dense can CentOS stream be used in a production environment? Totally. Absolutely. Many of us do use it that way. There's been a lot of, I'll just call it a lot of fun around that. A lot of people try to say like no you can't use it now it's too unstable, it's this totally different thing but the reality is if you actually look at it the difference between RHEL and CentOS at any point is like 10% like it's 90 to 95% the same software versions may have a few extra patches things that are as you can see in the diagram we're a little bit further back in time here but what happens here in CentOS is what's being planned to go into RHEL 9.2. So it's not just experimental we have that on the not slide here. Yeah it's not bleeding edge, it's not experimental it's RHEL style updates, it's things that the business people want to put into the next RHEL minor version and breaking compatibility would literally break contracts with customers so that's not what's gonna happen not that mistakes can't ever happen they certainly can but it's not just throw this against the wall and see what sticks there's QA and stuff happens before it goes into the CentOS release so I think it's great talking about using it in production you get bug fixes before RHEL gets them and a lot of security fixes as well so I think it works a lot better in that use case. One thing I will point out to people is that the traditional CentOS users that would use CentOS in like the first five years of the lifecycle that care about those new features coming in the minor versions are a lot better they like CentOS Stream the new model a lot better than the people that would use CentOS in the years five through 10 where they want it to not change at all there's no more minor versions those people and they want the 10 year lifecycle as well which CentOS doesn't have anymore so those kind of people there's gonna be a preference to something like RHEL or something that tries to copy RHEL but CentOS for people that were using it in the first five years it's still a great operating system still the same major version very similar software and works most of the same way it's very functionally it's basically the same it's just the development works a little bit different speaking of that development do you want to talk about contributions? Yeah sure and I'll add to that I think different people have different production environments right so for some people like the those minor release branches are very important and for some people they're not like if you're the kind of person who would who would have gone straight from nine four to nine five as soon as it comes out then CentOS Stream is you know perfectly fine and if you're the kind of person that would redo your system after five years which a lot of people do but some people want these 10 year life cycles minor stable minor branches or whatever ooh fancy so it just some of it depends on what your needs are in a production environment okay so CentOS Stream I think we kind of actually address a lot of this a little bit we've hit around it danced around it but some of the issues that we had with CentOS as a rebuild project as a clone is that you couldn't actually do anything right it was people like to say it was bug for bug compatible with RHEL and so if you were to file a bug against CentOS the rebuild saying you know this package is broken or my application doesn't work in this environment or something if you could reproduce that bug in RHEL well then it's not a bug in CentOS right you need to go talk to Red Hat and get it fixed in RHEL first so with CentOS Stream we're actually able to fix bugs in CentOS as they're happening and in fact that's how RHEL bugs are fixed like if you file if you're a RHEL customer and you have an issue you probably have an account wrap or whatever I don't work on the RHEL side of the house so but you would talk to them but it's all gonna go through CentOS Stream me neither y'all aren't talking to sales people yeah I'm not a RHEL sales people I'm a CentOS promoter so and as someone and I felt that pain a lot in the project in the early pre-stream model and everything I felt that pain I would file a bug with the project and then run headlong into like yes we see the problem we reproduced it on RHEL so we're gonna close this bug because our goal is to be bug for bug compatible that is a frustrating experience from the outside from the contributor, the reporter, whatever it's also a frustrating experience for the maintainers on the inside which later I became for a while for a few years I was working directly on CentOS at work and it's just a frustrating thing all around where you're just okay you can use it 100% as is you can't really you can do your own changes to it but you know you can't get anything into the distribution for the greater good for everyone else which that's kind of we kind of we fixed the glitch in a way now moving upstream of RHEL CentOS can accept those contributions the RHEL maintainers now maintain their packages in CentOS also and to fix it in the next version of RHEL the next minor version they work in CentOS do that everyone using CentOS gets that change first there and then it shows up for RHEL customers later RHEL maintainers also have the option to do stuff in older minor releases but there's a lot more red tape for that and that's frankly it's off top for this but it is sort of tangentially related Oh, I have this as a sticker sheet I'll put it out with the cookies later I'll pass them out Oh, go back real quick? Yep No, that's a clicker for the size of it Oh One more Yeah, not just the fixing the bugs ourselves you can fix the bugs you can actually do things if you could report a bug and also say I found this flaw I identified the issue upstream here's the commit that fixes it and here's a pull request to add it as a patch to the package and you can do all of that end to end and the RHEL maintainer can just come along and say yep, this all checks out and we should fix this we should fix this in RHEL and in CentOS, merge and then they do their work to get it in the product optionally into the older minor versions also but CentOS maintainers who are now RHEL maintainers are unable to actually merge those contributions from the wider community rather than just closing down stuff as you can't change it because we have to stay the same I want to pass these up Cool So that's CentOS Stream which is, as I said kind of the core operating system that everything else is built around but I want to talk more about kind of the rest of the project because CentOS Stream is a good operating system it has a lot of good uses but I think some of the more exciting stuff happen with what we do elsewhere within the project and so first of all we have actually a board and the board is to set strategic direction for the CentOS project it is not supposed to be a technical governing board except it keeps getting kind of bogged down in some technical things because we don't have other ways to deal with them which I'll address in a second So it's 10 members they are kind of self-appointed whenever there's an opening we have an open nomination process from the community and then the board votes on who will fill that we actually just went through this process we had a board member step down and we just went through the process of appointing a new one so board members are able to serve for as long as they're able to serve people come and go and get busy with other things in life so all of the board members except one serve as individuals regardless of who they're employed by so if you're on the board as a Red Hat employee you happen to be a Red Hat employee but you are supposed to be on there speaking for yourself and not speaking for your employer and we have people on the board as well who are not Red Hat employees the one however is Brian Exelbeard who is the Red Hat liaison and his job is actually to act as the voice of Red Hat to the project so any of the other Red Hat employees on the board are not speaking for Red Hat it's the voice of Red Hat Brian with his glorious beard speaks for Red Hat but he doesn't have a magic veto power the whole board operates on a consensus model so I mentioned that the board keeps getting bogged down and like these technical issues and so a proposal that's going to go through soon and it's been through a few iterations here is a SIG council and this is basically a way for all of our special interest groups to speak to infra to speak to the stream engineering team and to have their voice heard and to talk amongst each other is to say oh we've been having this problem in our SIG or this problem over here and to work together to solve some of their common issues and this would be made up of a chair from each of the SIGs as well as people from stream engineering and rail engineering and they'd have regular meetings to kind of work through everything so as I said it's an open issue but I want to bring it up because it's I think actually a really important development in the project giving us more of a technical board allowing the board of directors to be a non-technical strategic thing hey Carl do you want to talk about Apple Apple is not part of CentOS but I'm going to let Carl talk anyway sure so I think I mentioned this earlier on in the in the intro about working on Apple Apple I'm not saying Apple a lot of people think that with my hiccups and I'm saying Apple like the hardware like the iPhone but no Apple or Apple extra it stands for extra packages for enterprise Linux it is part of the Fedora project not technically part of the CentOS project but all related and friends and directly works with CentOS and I'll talk about that in a minute well right here on the last bullet Apple basically is Fedora packages that have been built to target CentOS Stream and REL which is a little easier to explain with this little diagram I made these are little boxes to represent packages it is not drawn to scale there's way more than like 100 packages in Fedora like I think like 60,000 last time I looked at it but around I think like 6,000 of them or so well basically the slide earlier where I talked about branching off from Fedora that's not all of Fedora it's a subset of packages basically the things that Red Hat wants to put into REL and support for the business for customers that subset goes into first CentOS and then into REL everything else in Fedora is eligible to be built in the Apple repo and it basically just shows up like another Fedora release if you're a Fedora package maintainer I see a couple of them in here that I know you have your package and you have like the Fedora hide branch at the front and then Fedora 40 is the one we're working on right now that's its own branch there's a Fedora 39 branch and so on Apple just shows up for the maintainer as another branch of Fedora even though it's not but it is so that's how that works it's a way for you to maintain a package in Fedora and then say I use this on Fedora on my desktop but I also want to use it on my servers so I'll request an Apple branch build it ship it there and then it's built to be compatible with REL and CentOS there's actually a really cool talk I would, I'm not gonna take up too much time in this because this presentation is not about Apple specifically but the CentOS Connect we did at Fosdum a month ago I have a talk that's on the YouTube channel about that where we're talking about the changes we're gonna make for Apple 10 and if you've used Apple before or heard of it or interested in using any Fedora packages on REL or CentOS I would encourage you to go look at that we're gonna do some interesting things because of a few different technical challenges we created a thing called Apple Next to work with CentOS Dream but we're gonna flip that model on its head in Apple 10 and Apple is going to immediately target CentOS 10 and then we're gonna have minor version Apple branches to target the REL releases so that way CentOS is a first class citizen for Apple packages rather than an optional afterthought that you might have to build for so we're gonna integrate it deeper and make it work a lot better I think but yeah, look on the CentOS channel on YouTube for that would an Apple 10 roadmap or some title like that? Yeah, I'm like that. Apple 10 overview, that's the title so. Is it meant to be a supplement to RPM and YUM or just specialized for packages? So Apple is RPM packages, 100% supplement, I would still say it is a supplement because they're designed by in the name extra packages, they're additional RPM packages for REL and CentOS things that aren't in the base distribution the first example that comes to mind my mind isn't a valid example anymore but in the older days, I think REL 7 the only web server in the main distribution was Apache, the Apache HTTBD server Nginx was not in the base distribution and you could get Nginx from Apple that changed at eight they added Nginx into the main distribution so it's not a good example anymore KDE is a good example GNOME is the only desktop offered in the base distribution but the Fedora KDE SIG also maintains KDE in Apple for REL and CentOS users that's a great example, old Nome Yes, does that answer your question? Apple will take on the status of primary and the RPM and YUM will go away or are all three gonna be active? Not as long as I'm working on it I am an RPM packaging nerd I like RPM Apple has always been about RPM packages a lot would have to change for Apple to not be additional RPM packages like so much that I don't think that's even feasible I know that there's a lot of move towards what people call like the atomic stuff from Fedora, the immutability but those are still built out of RPMs and then there's RPM layering and other things like that RPMs not going anywhere and Apple only has RPM packages Any other questions? Damn you For the RPM packages that you're talking about are you also going to include modules in Apple for 9 and 10 or is that not a I already know the answer to this Jim Ringer So the question was about modules If you might have seen it before in Fedora modularity You came in at just the right time, Neil Oh, I know Modularity Oh my God, we're talking about that? Yeah, so it was Jim that asked the question so he's trolling us lightly So there was another technology it was still based around RPM the idea was that you could have parallel versions of RPMs in the same repository and through a DNF command configure which set of those RPMs you were going to look at kind of like namespaces on a repository The idea, I love the idea the implementation left a lot to be desired It didn't work very well We actually did have an Apple 8 modular repo for a while and but because of infrastructure problems we had within and other things and low usage we eventually decided to just shut it down and whatever, turn it off and so we don't have that anymore we didn't do it for nine ever and we don't have any plans to do it in 10 and sneak peak while rail 8 has a lot of modules rail 9 has fewer and rail 10 won't have any so that is going away that's an old technology don't really need to know that much about it other than people that have been inside the project for a while you can mention it to them to make them cringe Any other questions related to Apple? All right All right, so I mentioned special interest groups and I'm just going to run through a few of them I'm not going to give a status report on all of them They are all supposed to give a quarterly update we do a monthly-ish newsletter so every month is roughly a third of the SIGs giving a quarterly update so you can just watch the CentOS blog for those and you'll see what all the SIGs are doing but I'm just going to just a few of the more active SIGs there is a new integration SIG so I mentioned everything everything that we do all the updates that go into CentOS are tested they pass a bunch of gating checks and that ensures that CentOS Stream itself is never broken the operating system but you don't run an operating system just to run an operating system you run an operating system to run your applications and so, yeah, right and so the integration SIG has been working on doing further tests to test the integration of the operating system with various applications to make sure to make sure other people's stuff is not breaking with any updates even if the updates are non-breaking sometimes maybe, you know your application is depending on some obscure feature I'm pretty sure there's a next KCD about this so if you have critical applications yeah, if you have critical applications running on CentOS Stream you can work with the integration SIG to ensure that things keep working for you the way I'll describe that is rather than coming along after the fact and saying hey, you broke my thing because you changed, you know this maintainer changed a thing that they thought didn't affect anything in CentOS and RHEL and it technically didn't but they were building on it in a way that caused them to be affected you can actually build in your own tests into that test suite and say that, hey don't release an update for this package again unless my use case works so it can no longer just be you're getting it as is you can actually have a little bit of ownership and make it your operating system too and say, the way this works here is important to me please don't release stuff that breaks that and I've got just some of the recent accomplishments I'm not going to do a readout of these but I asked some of the SIGs to provide some of this stuff hey, let's talk about the high for scale SIG hey, Neal oh no, okay Michelle that's right yeah that's okay yeah, it's okay so how do I put this thing on? you want the handle? yeah, I want the handle I don't want to figure out how to put this on all right, so this is where I need to stand? yes okay I hope so too so the hyperscale SIG did not expect that we do stuff around I'm one of the leads for the CentOS hyperscale SIG we do stuff around enabling CentOS stream deployments and large scale things like whether it's huge fleets of desktops and laptops or even larger fleets of servers and cloudy things and containers and all the whole things as far as that we've done a whole lot of things in both CentOS Stream itself contributing to the operating system again, as Carl mentioned earlier it's about making it your own operating system not just by adding tests to qualify that your stuff works the way it's expected but also to change the stuff that's in CentOS Stream to benefit you and everyone else so we've done some stuff here if you're familiar in the Linux space about how Linux audio is actually shot up to be the best in the world that's pipe wire and wire plumber and we did that along with making it pro audio work out of the box in CentOS Stream 9 and of course, RHEL 9 and with SIG content we do things like provide back ports of various packages including notably system D we provide Fedora based kernel to add additional features that people wanna do in the desktop and servers, container images, expanded virtualization and cloud images and all this other stuff all in service of being able to support our goal of providing people a high quality experience to being able to use CentOS how they need to be able to use it or want to be able to use it but I would say so thanks, I love talking about the hyperscale SIG because it's basically entirely non-Red Hat yes, well the one person okay, but they're just like okay, we could take CentOS Stream as a base and make something else that we want because sometimes there's stuff that maybe we don't wanna put in CentOS Stream but you could make a very different system in hyperscale SIG and then some stems sometimes some of those things do end up in CentOS Stream, which is nice there's a Kmod SIG which packages a bunch of kernel modules again for stuff that maybe we don't wanna put in CentOS Stream for support reasons or whatever but they package up a bunch of kernel modules for things like different file systems or supporting various pieces of maybe obscure hardware but it's not obscure if you need it right so they build a bunch of kernel modules and they build them for both CentOS Stream and they build it for REL and those REL ones should be okay for any of the kind of REL derivative systems out there the alternative images SIG is another one of our new ones so we have certain images that we produce operating system images you can download and install the alternative images SIG has sprung up to kind of create other images you might wanna download so right now what they've made are live images they have a live image for Nome and then one for am I handing it back to oh we're taking a picture Neil is also involved in the alternative images SIG so but there's a Nome and a K2E live image so if you wanna run a live image there are images for that we'll have words afterwards but they're working on a lot of other images too so you could imagine say images around that already have certain other special interest group content enabled right off the bat so you don't have to kinda do that like the hyperscale stuff so you can get maybe a hyperscale image instead of doing CentOS and then adding hyperscale I see a few new faces did anyone not get the sticker sheet that would like one I'll come bring you one okay yeah we'll have them by the cookies at the end did Jim give it to Jim? I did I chased him down to give him one okay good all right almost done there's an ISA SIG this is also a fairly new one and this SIG is investigating new CPU features so the x86-64 architecture gets new instructions and stuff from time to time all of the CentOS stream packages right now are built on the V2 baseline and the ISA SIG is currently investigating basically building packages with a V3 baseline which allows it to use additional instructions in the x86-64 instruction set which potentially can mean packages that perform better and part of what they're doing is like okay let's build these packages with this different baseline and then run it through tests to see is there a better performance and is it worth switching to that by default which could maybe leave off some older systems sneak preview I haven't seen anything publicly posted about those results yet but talking to other people I've heard chatter that at this point using V3 as the baseline for 10 is probably gonna happen like absolute like a really big blocker that's like this particular customer is completely broken with that and needs us to be on V2 it's probably gonna be V3 for 10 sneak preview don't tell your friends are we recorded? we have an automotive SIG this is very cool so one of the things Red Hat is working on is an in-vehicle operating system and they do all of their work upstream in CentOS and so we have what's called auto SD which is a, it's basically a distribution of CentOS Stream but tuned for these automotive applications and that is the work that would feed into Red Hat's vehicle operating system but it's something that you can play around with too just in the community space and try to make changes too so oh, they gave me a diagram over there accomplishments I'm not gonna do a read out if you can see it. There is a cloud SIG they're involved with building both the stuff in the open stack and the open shift ecosystems so the RDO distribution of open stack gets built within there the OKD upstream of open shift which is a Kubernetes distribution gets built there and they also do, they build ASCAS which is core OS is a Fedora image based system but ASCAS is basically core OS but built from CentOS Stream. So one little parallel there I wanna point out for the automotive SIG what they all tell you a lot of times is that auto SD, CentOS Stream automotive edition or whatever you wanna call it is to CentOS a