In this section, we are going to discuss SystemD for administrators and in fact, we're going to chat about SystemD N-Spawn. This is a continuation of what we had done last conference and so the adventure continues. Let's go see what's up. Our standard header information and to our welcome screen. So welcome. My name is Lee Elston. I am a instructor and course maintainer for the Linux Foundation. I've been in this business since about 1978 and have had a various number of different jobs in all kinds of different organizations over the decades, decades. So now I do the course maintenance for a few courses listed here, the LFS 311, LFS 416, and LFS 216. So we'll have my administration look on what we can do with SystemD N-Spawn. This presentation is for administrators. In many cases, we have had other SystemD N-Spawn presentations created and they were targeted more for developers. This time we wanted to look at administration aspects of it. As an administrator, we can use SystemD N-Spawn. So what we're going to look at in the next few minutes is what is SystemD N-Spawn? How do I get started with it? How do I get my first machine running? How do I connect up the network? And what are some of the built in and optional services? And when we say built in and optional, we're looking at SystemD itself, right? So we can leverage the SystemD optional services to do things like networking and name resolution and other features that we need to get running. So we're going to use much of the SystemD infrastructure and ecosystem during this presentation and talk. The idea of restricting applications or services to a confined area in the machine is not a new concept. Chirute has been there for a long time and Chirute stands for change root so that you get to have a different root directory. So rather than the standard slash directory or slash root, however you wish to refer to it, we can have a subdirectory of, say, slash home slash testing and then we can create within that with LDD and some other programs, we can create a list of all of the files that need to be put into our Chirute environment so we still have to build it. And then once we have all those pieces in there, we can actually go and use the Chirute environment. An example would be if you're going to run SSH from inside the Chirute environment, you would have to go and discover all of the shared directories and shared libraries that SSH is going to use, hence the mention of LDD. So that way you can use the linker to find out what all the linked objects are required for the program and then you can put them into your Chirute environment. Sounds like a little bit of work, doesn't it? Well, yeah. And then as soon as you get it working and you've changed root into the environment, you're going to discover that Darn, I forgot something else and have to exit the Chirute environment and mix it up. But once you get it set up and once you get it running, it runs extremely well and it does restrict you to a single directory. Now, Chirute is very old, so it doesn't take advantage of some of the newer facilities, right? Like namespaces. We'll talk more about that. So virtual machines and proprietary containers, so other container manufacturers, not the open ones that are included with Linux, they actually work extremely well, right? They can use those to isolate. The most isolated environment is a virtual machine, right? You have a whole copy of your own operating system. But if you're just setting up something to test, you've still got to create the initial disk drive for your virtual machine. You need to have disk space larger, be careful, a larger amount of disk space, I'll say, because you have to contain the whole operating system. You are going to need to have your network connected, of course. And if you go the container route, then you have to have the container components that you need to build up in order for you to have the access to the programs that you want, right? So if you're going to do, say, a web server, you need the web server component image, and then you're going to need the little image that's the networking component, and you may need some other little pieces. And it stacks all up like a cake, and it works really, really well. The only thing is that it might take a little bit longer or more in-depth knowledge to set some of these things up. What we're looking for with SystemDnSpon is a very easy, simple way to create a container, right? We want to be able to create this container very quickly, very easily, and have it just work. As a creator of SystemD mentioned, the best part is it just works. Some of the reasons you might want to use a SystemDnSpon container are listed here. These are the reasons that we're given out for creating it in the first place. So use it for testing, use it for debugging, use it for profiling, and using it for building. The builders are the people who go off and create no distributions and new packages and stuff, and if they'd like to do it in an isolated environment, awesome. They can go do it inside of there. So still heavily towards building and perhaps developers. My part of it is I like it for testing. If I want to test an application, I would much rather put it into a little container, right? So I could put it into a SystemDnSpon container relatively easily, and then I can fire it up and away it goes. So that's a little bit of history as to why we might want to do this. So we'll go around and say, well, what is SystemDnSpon actually? Well, it is a lightweight container. All right, so if it's going to be lightweight, it may not have all the features and functions that a commercial industrial grade container has. It may have a little less functionality, but it also may be a whole lot easier to get started up and get it running, right? You don't have to have extra, well, there is one extra package, just one, but it's not too bad. We're going to see that actually later, so we'll have a chance to go through that as well. SystemDnSpon makes use of namespaces. Namespaces, if you haven't bumped into them yet, is a mechanism that we have inside of the kernel to duplicate, replicate, create new ones. There we go. Create new structures for confined use. An example of that would be a process table. We all know that if you type in PS-CF, you get a list of all the processes on the system, right? Assuming you're written. And you get to see everything. So if we put you into a process namespace, okay, we're going to hide you off in the corner and then you do PS-EF, all you're going to see is the processes that are related to you and your container. And it's a much smaller list, and we're going to see that as well. When we install a container and we're going to go have a look at it to see what can we see now that we have a container running. It does behave much like a charu environment. The installation process of the container is going to copy over some necessary files from the installation media so that it can function in the environment you've got. So if you've got a, we'll pick in a Buntu environment and then you're going to create a Ubuntu N-Spawn container, then you probably don't need a whole lot of extra components to go with it. And we can get these to install very easily. That's another part we'll do later. If you're going to do, I've got my Ubuntu machine and I want to load Fedora, then it might copy a few more packages or a few more files over when it builds the root environment for you. Okay. One of the things that I like System DnSpawn for is I can easily launch an operating system and get a login or I can have it just simply run an application. So I can launch the container, the N-Spawn container. I can launch it with extra parameters and actually it'll just launch, execute the application and when it terminates, it cleans up and it's all done. So it has a very, very flexible configuration. So depending on what you want to do, and we're going to go through some of these sequences here and then of course we have our demo coming up at the end that we get to see all those stuff in action. So those are kind of some of the things that we need to look at. Now, in this big container environment that we do have, and containers are everywhere and they're wonderful, a lot of personal preferences, right? But with all of the different virtual machines and containers, why would I want to use System DnSpawn? Well, for one reason, it's integrated into the System D environment. It does have its own package, System D containers is the name of the package, but it has its own package but it only has a few components in it. So it will allow you to do it very easily. Your distribution will have the necessary components required to get System DnSpawn running, not a problem. It's going to use the kernel facilities like name, space, and C groups. Some of the other containers may or may not use all of those things. C groups, right? We all know that that's going to allow us to restrict, restrict or assign specific resources to our container so we can do some performance testing and we can make sure that our container is not taking out over too much of the system. All the things you get to do when you want to do some performance testing, right? And you don't want or you don't want to have a specific container or a specific application or program consuming too much of resource. It's very easy to configure and it is extremely well documented in the man pages. When going through this, I discovered that, yes, the man pages are awesome. Although they do take a little bit of reading sometimes because they're wonderfully technical and correct. And I just got to get them into the concept of, okay, how do I get this for the administrator so that it's a little bit easier and we don't have to worry about some of these other little details. We have access to the details. That's not the problem. The problem is we want to make sure that we can get this up, get it running easily. Therefore, we can start to make use of it and then we can go back and extract all the little interesting details to customize it later. So the prerequisites for SystemDNSpon, not a whole bunch of prerequisites for it. SystemDNSpon is the container. It may be a specific package, right? Usually the package is called SystemDContainer. That's just usually what the name of it is, depending on your distribution. So the programs are SystemDNSpon, which is the main program that spawns off the container. We have a machine CTL, machine control program that allows us to interact with the SystemD machines. Now, SystemD MachineD is another program that will run and it actually runs it back around and it helps with image, import and export services. So if the machine CTL has been instructed to import a new virtual machine, sorry, into container, or export a new container, then it calls SystemD MachineD to do it in background course. No, we call SystemDImportVM to do it in background. Tarn. Anyway, we're all human, right? Anyway, usually the package is called SystemDContainer. Check your distribution. Having a network connection while creating containers is an advantage. And it's a huge advantage because we can use the resources of the repositories to fill in the blanks, right? So we can issue a command that says, okay, I want to build a directory style root file system, right? And it doesn't have to be a separate file system. It's just a file, right?  So a directory that looks like root. When it creates them, having a network connection is really handy because it can just go grab them off the net, bring them in, okay? This also we're going to have a look at as well. Speaking of, here's some places that we can get some of our image sources from. And this tier is a couple of ways of creating different virtual containers, right? How to create a container. So one of the first ones, this one is one of my favorites. SystemDNSpon-n-root directory-x-b. Well, what it's going to do is this is going to create a snapshot of the root host. And then it's going to say, fine, but I'm all finished, throw it away.  So the directory is slash, right? Or root. That's where it's going to get its information from. And off it goes. So this literally is going to create a snapshot of the host system and away it's going to go. Now, the host system, when we do these snapshots, they can sometimes be pretty big. So do some experimenting with them in your environment and decide whether or not you like the snapshots. I find them slow to start up. That's all. Other than that, they run fine. You could use a pre-configured or pre-created cloud image. Many distributions, if you go in and look at their downloadable things, under for Debian or Ubuntu or Fedora or CentOS, many of them will have a cloud image that you can download, right? And it could be a type QCOW or it could be a QCOW that's compressed. But you've got this pre-configured image. Now, some of these images, I'll mention this here, some of these pre-configured images think that they're actually going to be loaded on one of the big cloud environments, right? So because of that, it may have a program called a package called clouded-nit. And if we run clouded-nit in our container, it's just going to fail. So what we're going to do is we're just going to remove it. We'll still be able to boot. Yeah, we'll boot. We may have a couple of little errors. And then we can clean up the errors by moving out to clouded-nit. Do we have a network? We don't have a network. So we'll probably get some other errors that are related to not having a network in our actual container. But they're easily fixed, easily fixed. So here we have our command as machine control. So it's going to do an import. So he's going to verify it. No, he's going to do a poll of a raw. So we're just going to take this and throw it on the system. And he's going to go to HTTPScloudcentoes.org. He's going to go look at CentOS version 8, 64-bit image. And he's going to go fetch the CentOS 8 generic cloud 8.1, 1, 9, 1, 1, 2,000, and whatever 13. x8664 Q-Cow. And he's going to call the new image CentOS 8. Cool. That's it. So it's going to grab that information from that Q-Cow image. It's going to put it all into CentOS 8. And then we'll be able to boot the machine. Excellent. Now, if I was on a Debian-type machine, in fact, I ran this on my Ubuntu machines. And it works just wonderful. And this basically says, I'm going to use a program called Debian Bootstrap. OK, now, Debian Bootstrap is going to be for the Debian Ubuntu folks, the other ones for other architectures. But for now, we're going to use the Debian Bootstrap. And it's going to go look for the repository unstable. And it's going to grab all the information from there. And it's going to store it in this directory. It's going to create a directory called Debian-tree in the directory varlibmachines. And it's going to put all the root information in there so that we can start the machine up. That directory, by the way, varlibmachines is also the directory that the machine CTL program likes to look at. So that's its default location. Of course, you can change everything and you can put it in your home directory. That's not a problem. But this is the default location if you want to use the machine control program. So there's another one. Just for fun, I had a Ubuntu 1804 image around. And it was just a QCOW image, just an image. Actually, we ran it on something else. It's a virtual machine image. And I said, machine control import raw. Bring it in. And it brought the image in. So the answer is yes. The question is, can I use old or can I use existing QCOW virtual machines or virtual storage devices? Can I import those? Yeah, you can. You're going to get a couple of errors, of course, because we don't have a networking setup right now. But yeah, you can. So can I use other formats? Yes, you can use other formats. And the format list, I'm going to refer you to the man page because there's a lot of different formats that it will support. Maybe that would be a option for a future time is to see how many different image types we can get to work. But for today, we're just going to install pre-created cloud images or directories from Ubuntu. Just because we're going to do it, that's the way we want to start off. I want to make sure that it works the first time, and then we can experiment. The dash x, I kind of mentioned in passing that the dash x was going to throw away that snapshot. The thermal snapshots. So you can create a temporary snapshot of a running system and running in a container. And when you're all done and you terminate the container, bye-bye, it goes away. So this is the least amount of overhead because I'm just grabbing all the ones for the local directory. That's fine. And the other part of it is that at the end of the day, it's gone. So that's an option if you're interested in testing with that. Always a good place to start. But as I mentioned earlier, the snapshot, when I tested some of the environments, it seemed to run a long time. And it's not that it was broken. It was just that it exceeded my patience. The cloud image, many distributions provide the cloud image. Absolutely. And some extra components are added for the specific cloud support. And you may wish to remove those, such as cloud init. OK. The system dnspond does not need extra packages. Just a straight up image will work. And the machine control, if you're using it, it can convert images to type raw. Because this is the kind of format that we would like. It would be the raw format. When we create that directory, the directory is going to become an OS directory tree. So this is going to be the differences of the OS directory from the base system. So we're going to need utilities. And we saw dev bootstrap already. We said dev bootstrap. And then we gave it some parameters to go create a directory tree for us. The DNF for the CentOS and Fedora folks can also do it. You just redirect the output of where the OS directory tree is going to start. And it will happily put it into a subdirectory in varlib machines, give it a name. If you're using Arch Linux, there's a thing called Pacman and Pacstrap. And there's also some other files. If you are interested, the little bit of investigation will find them for you. And you'll be able to see what types of images you can use with your directory tree. So this directory tree, it is going to be literally the operating system root directory. But it's not going to contain duplicated binaries that we can have already from our base machine. So it's going to be much smaller. When you create a couple of these, go back into the directory, right? Go into varlib machines directory. And check and see how large the directories actually are. And then do it for the same OS as the base, do it for a different OS on the base. And you can see how much space it's going to consume in each of those environments. So that's where it's going to hang out. Other OS images, yeah. How far do we want to go down the rabbit hole today? You can go several things down, right? As far as other images go, you can use a converted virtual machine image. And here's an example of one that we did, right? It's an 1804 Ubuntu image. And then did the machine control import. So we said import, yada, yada, yada. And then we told it to nspawn. And here's an example of the nspawn, system d dash nspawn dash b. So it's going to boot up an operating system. It's supposed to run it up into what looks like run level one, right? So yeah, we're going to have a regular initialization sequence. And then we're going to use our file is going to be called, sorry, our directory is going to be ubuntu18-04.raw. So you see system dnspawn will, by default, go look in slash far live machines. So you pass it just the name of the machine. And that actually is the directory name. When you pass it there, it'll just hop in. If you wanted to redirect that to a different location, you, of course, can do that. Just put the image someplace else and change the location. In this case, we actually added a bridge. Here's our first look at a network option. It's command line option to system dnspawn. We said, hey, build me a network bridge. OK. And by the way, I have a network bridge already called verber0. And verber0 is the bridge that Livevert creates. So this machine has Livevert installed on it. So I'm just going to hook onto that bridge because I know it's going to give me an IP address. It's got a DHCP server on it. It's got an IP address. And we know that it's bridged to the outside world. So I can hook it up in a way I can go. I like using Livevert. The reason that I do is I can hook up to things like wireless devices pretty easily. So I just feed all the parameters in the Livevert. It does its magic in a way it works. So Livevert builds my bridge for me. So I just build, sorry, Livevert builds the bridge. We just connect to it. And if I wish to have different types of bridges, different types of network bridges, then I can go into Livevert and change them. And I usually get into Livevert through Vert Manager. So you can go through there and create different ones. Some are bridged, some are routed. You can have all kinds of interesting networking constructs that can go on there. Or you can build them yourself. If you decide that you'd like to build a bridge called BR0, awesome, go for it. You can create your bridge in your taps and you can connect them all up. We're looking for a way to make sure that we can get this running with the least amount of anxiety. So just let Vert Image do it, right? Let it do it that way. And then that way, sorry, Livevert. Now Livevert has got it running, then away it goes. So that's an awesome way to do it. And if you prefer to do it by hand, by all means, let's get it working first. Then we can go test it and change it. When you want to create your first InSpawn container. OK. Absolutely we want to create our first InSpawn container. So our command that we've used is debboot unstable. So we're going after our Debian machine again. We're going to the repository called unstable. And we're going to put it into slash var, slash lib, slash machines. And we're going to call it Debian Tree Image. So we're going to call it. And we can let that run. And you're going to find that a whole bunch of stuff starts flying by on your screen, including a bunch of things to say retrieving, package doing, validation of package. And it goes on and on and on. And it runs for a little while. A little while, a few minutes. Hard to say, depending on your network, right? Because this is running out to the network to go get it. So it's going out to the network. When it's all finished, it'll say base system installed successfully. And I'm confident that it'll say that. IMF is very confident it's going to say that. When we're all done, we can use the machine CTL list dash images to verify that the image directory exists. So list dash images will list all the images. Machine control has other options, too. Yes. Machine control list will list all the active machines. Anyway, we'll get to it later. So this is your first container, right? We've got to start it. We haven't started it yet. We downloaded it. Now we're going to start it. And we're going to start it with system dn spawn dash m. And that's the machine name. And the machine name is Debian Tree Image. All righty. So off it goes. As soon as I hit that command, it's going to come up with spawning container Debian Tree Image on that location. And there may be a few other little lines of output that come back. And it's going to give you this ominous little press. The shift six, the little hat, open square bracket, three times to kill the container. So you need it to do that. So correct, bracket, three times, away you go. Now, what state is this going to be in? Well, this is the text that comes out, right? We typed in the command, system dn spawn. It replied with spawning container. Press three times. And then it responds with root at Debian dash tree dash image and give us our user-friendly root prompt. So the machine's there. That's it. We can log in, right? We can tell it to do things. It's given us a root shell. Now, that root shell is process ID 1, OK? It's like, oh, yep. So because of the way that we did this, that's process ID 1. So no init processing was occurred. No additional system d processing was occurred, right? It is just simply it came up. And our init program is slash bin slash bash. So there we are. Now, the fun stuff. What can I see? OK, so I've got my machine. It's up and running. I have a root prompt. What do I get if I type in ps-ef as root? I get to see only the processes that are related to the container. It's going to be a very small list. And it's like, oh, OK. I will notice that process ID 1 is the shell. I'll be able to see it. It's right there. PID 1 and then be bash, OK? So it kind of works like single user mode or emergency mode where you put in init equals slash bin slash bash in your grub line. It's going to work like that, OK? So it's awesome, except can't do much with it, because I want to have some other support services to go with it. But I can do something extremely important at this point. I can set Roots password, because he doesn't have one yet. So I want to set Roots password, OK? That's a good plan. And in my case, I usually add myself. I just like to have myself added it right away. I have two user IDs. If I wish to then have a look around at the networking subsystem, if I did, oh, I don't know. Netstat-I, oh, I know. Just simply IP-A. Just go look at your interfaces.  You're going to find out you can see all of the interfaces on the system. And then you're going to come back and go, hey, Lee, I thought you said that we're going to have these isolated. And the answer is, it will. As soon as we give it an adapter to isolate, right now we don't have one. So it's going to have access to the original networking stack at this point in time. So IP-Space-A will show all of the networks right now.  But it's the root file system under the container image, not under real root. So these are some of the things you get to see. Now, what else are we going to do with this? I mentioned we're on process ID 1. Process ID 1, as a bad shale, is pretty cool. Although there are some considerations of being process ID 1 that we don't generally fuss with too much. So it's probably not the greatest thing to run things as. So if you have a non-init-aware program, probably not a great thing to run. My suggestion here would be, if you're going to do it, if you're going to run as process ID 1, then just do the minimum required, get out of it, and either become process ID 2 or bring the machine up with the init process as well, which we're going to do in a minute, OK? Advantage of process ID 2. If you run the machine as process ID 2, and you pass it an option, by the way, it's command line option, did we put it down there? No, we didn't put it in there yet. If you put it in as process 2, that's A does it. Then when you do that, it will look for a program to run, and it'll install a stub init process as process ID 1. So there's just a little stub init process. And then you can run whatever program you want as process ID 2, OK? So it works out pretty well because I've got my process ID 1 as the stub. Now I've just got a regular process running as process ID 2. I can run it for as long as I want to, or I can pass it a command. And in the case here, in our notes, we sent it a command. We sent it the command IP space ADDR, right? So we started the machine, we looked at it, we said, show me your devices, and it can terminate it. Because when that program terminates, bin IP, then the machine will terminate as well, OK? So it does give you a great way to run. So that's process ID 1 and process ID 2. We need to do some more, don't we? We need to do a normal boot, a normal boot, meaning we're going to bring a machine up and we're going to run an init process and fill in some of the blanks. So our command, it got really small to fit on the page, sorry, in spawn dash B for boot, dash M to tell me what the machine name is, press the Enter key, and you're going to see your user-friendly startup stuff, right? You're going to see the init process starting up. So our good old buddy systemd is going to do all that stuff for us. I mean, you can see, I created a slice and it dispatched this and it sent the forward and it located an encrypted volume and it did all these wonderful things. And at the end of the day, it said Debian can do Linux bullseye SID on machine lake. We put the machine lake, that's the machine name, console and a login prompt. Now, fortunately, we've already set Roots Password. So we can log on as Root now, or log on as your other user if you created additional ones. But the cool part is, we're now up with the init process. Will this run perfectly with no errors into OK section? Well, we didn't start a network, so I expect it's going to complain about a few things. So you can scroll back up the output and have a look at what init processes were unhappy with your current environment. And then you can do what you want to fix them. So an example might be cloud init. Cloud init tried the run. OK, fine. Remove the cloud init program. You don't need it. So just erase it. Anything else that you don't need is trying to get in there. You could possibly erase as well. But then remember, we still don't have a network set up. So even if you do have some errors, the errors may go away if there's already a network. Because some of these processes, right, these init processes say after networking. And well, there's no networking head. So whoops, no adapters yet for networking. So we do need to get things just a little bit sorted out.  You're good. But most of us like to put some new program on here to test. So you may not have put it into the directory manually. You may want to do it this way, like a normal way, right? That's the same thing over. Let's look at some network options. Because I thought network is a good thing to do next. There are several options for networking. In the man page, system D endspon. Chapter one has a whole list of these. It's a great thing. So there's private networks. It disconnects itself from the whole networking subsystem that's on the host. And it'll have a private network on its own. It can have a network interface. So you could assign a network interface to your client, to your container, your client container. And then it owns it, OK? It's got some of our friends, Mac VLAN. Mac VLAN for additional MAC addresses, the IP VLAN as well. The vEath adapter is there, as well as the vEath extra. And then I've got my brace that I used. It supports network zones if you're using Fireball D, right? So you can have a network zone. And if you really want to specify the network namespace path, you can. You can put it into a location of your desire, right? Wherever you want to put it. This is one I like, port. I can open a port to my specific, a specific port to my container. So it gives me some other options, as well. So all these options, the dash dash options, all go out of the command line. And we can just select them as we will. Notice the format. I like the format. The format's pretty cool. We're going to hash dash dash name, right? And then if it has an argument, it's equals and then equals and a value. So it's kind of neat. I think the equals sign got eaten off the port. But IP VLAN, for example, IP VLAN equals. And you give it the VLAN number. In the Mac VLAN, you give it the adapter name. Yes. So the options and the output and the connection is all documented in the man page. But those are, it's pretty impressive, a little list of network options. So let's have a look a little bit deeper into some of these. The network bridge, we selected live vert because it was already active on the host. That was it, right? We wanted to concentrate on N spawn. We didn't want to concentrate on figuring out the network today. So the dash B is the boot. The dash M says this is the machine name. And the network bridge is going to be verb zero and then away it goes. By default, the image that we used didn't have the networking turned on. There was no network. So there was no network manager. It wasn't running. And system D network D was turned off. So let's just turn it on.  You could enable it so that it stays enabled if you wish because we do have a rewrite environment. And what it's going to bring up is system D network D without any configuration. It will bring up a loopback device and if it finds another network device, it will try to assign a DHCP address to it. OK. Our DHCP address should get assigned to the bridge from Livevert. So it should get in from Livevert. So as soon as it comes up, it comes active. It says, oh, look, an adapter. Livevert says, would you like an address? So our network bridge is going to have a Livevert address on it and it'll be wonderful. One of the other options you might wish to do is you might wish to run N spawn as a service. Because so far we've done this all manually in the command line. And that's a great way to run them because typically if you're testing or experimenting or doing something, you're right there anyway. So you can just start them up. But there is a system D service for controlling N spawn containers. And it accepts the container name as part of the startup command. We talked about this in earlier sessions where we can have a variable being passed from the system D service right down into the service configuration file. And we can make use of the information that's in that variable. Same thing here. We're going to use the variable. And the variable is going to be the name of the container that's in barlib machines. So mine's called Debian Tree, right? So I'm going to say system CTL start and then system D dash N spawn at Debian dash tree dot service. And it's going to bring it up. That was probably some stuff that we've got to do because we had a big command line list of options, right? And how do I get all of those command line options into my system D service? There's a config file for that. In the service configuration file, there we go. Now, this is a configuration file that's in a specific location. It's in ETC system D N spawn. And by default, it doesn't exist. So you have to create it. And then when you create it, you also create a file inside that directory that is the machine name, right? So in my case, the file is going to be called Debian dash tree dot N spawn because it ends in dot N spawn, not dot conf dot N spawn. And it's going to go into the directory ETC system D N spawn. So that's what it's going to go. And then I can fill in the content that I want. Now, I'm going to take defaults because that's what I've been using so far. And then I've been adding a few extra things in passing. So I create an exec section and say, yeah, OK, I want this to boot, right? So boot on the dash B option. That passes the dash B option, OK? I did some experiments with process two. So I thought, well, I'll leave it in here as a comment, right? Process two equals on. So I was, oh, yes. We can run a process. So when we go, because we're doing a process two, then I could actually pass another parameter in here being the actual command I want to run. So I could have put it in there, but I didn't. I passed the hostname. I called it Lake. Just because I wanted to pass the hostname to it so that you could see it. If you don't pass the hostname, it's going to inherit the hostname of the base machine. So just be aware of that. And it does sometimes get a little bit confusing when you're on the text-based command line, because the prompt will say the base machine all the time. And you could be on the container or not. So it is helpful to set the hostname. And yes, you can set the hostname on the command line should you wish to. Now, on the other network section, I decided to say the bridge is going to be verb or zero. So that way I don't have to type in anything else. So with these parameters, I should be able to start and stop my virtual machine as I want to. Excellent, excellent. OK, this is our summary. So this presentation, we discussed the following aspects of SystemD and Spawn. So what is SystemD and Spawn? What is SystemD? Well, SystemD and Spawn, how to get our system services started, our SystemD and Spawn services, and configuration files and directories, built-in optional parameters, et cetera. Now, one of the optional services that we did use was SystemDNetworkD. We used it to start the machine up. And that concludes our chat part of this session. I would like to thank everybody for attending. I hope this was of use. The examples that we had run through here in the lecture component of it, we're going to go through in the lab component of it. And I'm going to demonstrate all these features and functions on a real machine. So I'll do these on the real machine. It'll be pre-recorded so you get to watch the recording. Nice thing about pre-recorded, you can download it again, which is cool. And the lab exercise will be available for you to download. I'm not sure the download link yet, but when we get around to the question and answer period in the live environment, I'll have that for you. Thank you very much for attending. Hopefully this was useful. Please enjoy the rest of the conference and be safe. Thank you. Welcome back. This is the continuation of our SystemD, the Adventure Continues, using SystemD and Spon. In the first section, we went through some of the lecture material to find out what the names were, how things worked, and we chatted about all that. In this section, it's all about the demo. It's all about showing you all of the things that we just talked about. This is all done and recorded so that you'll be able to take it home and try it on your machine. And hopefully you will have as much success as I do with this. So let's get underway. We need to check a few things. We have our nice yellow screen here, which is my default color. On our system, we would like to check to make sure that the prerequisites are installed. This happens to be a CentOS 8 system, so I can say RPM minus QA pipe-grap SystemD. And you'll see that I have several SystemD packages already installed. I have SystemD container, as well as the basic SystemV239, and some other ones, PAM and Python and libraries. But it's the SystemD container that I'm most interested in because it has the extra programs for us. Another program that may be useful that, since we are on CentOS 8 and it does have access to EPEL, we should have another program, which would be kind of fun to use. And from our EPEL, we have the Debian Bootstrap program, Deb Bootstrap. So we can use the Deb Bootstrap to create ourselves a OS tree directory. Excellent. That ought to be good. And we'll do some other stuff as well. So let's get on with this, have a look, and see what's happening here. So first thing, we need to go get ourselves an image. Well, I really like the Deb Bootstrap one, because it's a nice line. And I am apologizing for cutting and pasting, but I'm a horrible typist, especially while I'm talking. So I made a file. Here we go. Look how quick I can type. Anyway, here we go. Bootstrap, Deb Bootstrap, we're going to go to the unstable resource, where we're going to get to go. And then we're going to go have a look, and we're going to store the output under slash far lib machines debi and dash tree. So out of the unstable repository into our directory. So let's start this up.  And it's going to have a really good time. While it's having a really good time, I think we're going to hop over to a window. Let's pick the next one over. Oh, look at that. It's plain. And we're going to go retrieve a cloud image from CentOS. Now, lots of typing in this one, but we can do it seamlessly. So we're going to use the machine control program this time. So the machine control program with verified turned off, we're going to issue a command called pull raw. We're going to see some more commands with machines in a few minutes. And where are we going to pull it from? HTTPS, Cloud, CentOS, or CentOS, x8664, images directory. And I would like to pick the image CentOS 8 generic cloud. And that's the one that we're going to go with. So the one we're going to go with was created in 2020, 0113. So we're going to let this go and start this up. Now, it says to press Control C to release your terminal because it's going to run a background. And ignore it. Let it run. So let's go back to the first one. Well, it's there. The first one has completed. So the base system has been installed successfully. What does that actually look like? Well, let's go look. In the directory varlevmachines, we now have a CentOS is already done, a Debian tree. Now, the Debian tree is actually a OS tree. So if we look at it, it looks like an OS tree, right? And it has the specific programs that it needs that are going to be different from the environment that is running on my machine. And we're actually going to generate a warning in a minute. So that machine is ready. So it's there. We can see it. It's installed. I'm pretty sure that one of the first things we want to do is that now we've got a machine is, why don't we start it up? All right? I think that's a good thing. So let's get this going. Right. OK, so in this case, we're going to do SystemD Nspawn. And what's the minimum amount of stuff that I need to put in here? Dash M Debian tree. That's the minimum amount of stuff that I need to start a Nspawn virtual container. Ta-da. I'm there. It's done. It's running. I wasn't very much grief at all. However, there are a few little bits and pieces about this. So first of all, we can see that we have to press Control Square Bracket in order for us to get the console back, because this particular console is over on Debian tree. Yeah, OK. So I need to press Control Square Bracket three times within one second for it to release and get my terminal back. OK, I'll remember that later. My user ID. I'm logged in as root. Now, we didn't specify a whole lot of stuff in here. So we've got the default configuration. So what processes do we have? We have us, OK. Oh, we only have two processes in our process tree, right? So our process table is part of the namespace. So we only get to see our own stuff. And you can see, by the way, that the bash shell is running as process ID 1. Yeah, it works. There is some process ID 1 considerations. I would only use this to set things up and then change it. I really wouldn't run too much as process ID 1. It's a little different. My PS tree, my PS minus EF, ran as process ID 4. So that's pretty cool.  OK, we assign a password to root user that we can log right in as root user. And we're going to get asked for a prompt in a few minutes. So let's do that. Password, the new password. All right, so I've got my guy. I can actually add in another user if I want. And here we go. All my full day of good stuff, wonderful things. All right, just like we expect. Just for kicks, we can have a look at that ETC password. And I've got a few things on it, not too, too many. This ETC password is private to this machine. Notice that Lee is user ID 1,000. Unfortunately, that is true across most of my machines. But you're missing all the other users. So this is not the ETC password that my system is running on. This is a different one. How about the network? Well, that kind of looks like about seven adapters, right? Number two, number three, number four, number five, number six, number seven, and all kinds of various things going on inside of there. Well, because we didn't specify a network when we started the machine, we just defaulted to the base machine. So we're looking at the network stack of the base machine. When we add in our own network, we'll only be able to see our own network. Now I think that that's something that we should probably go do, right? So just go add in our own network. So let's do that. So I'm going to stop it. Two ways we could stop it. Let's just do the control, square bracket, one, two, three. And it comes back to the machine grumpy. Grumpy is my machine. And now we're back here. We should be able to relaunch our job. So first of all, let's not do too many things at once. Let's say dash A. And if we look up dash A, what it says is it says we're going to do PID 2. PID 2? Yep. All right, so we're back again. That was pretty quick. But we had an extra parameter. So if I do PS minus EF this time, I find out that there is a stub and it processed as ID 1. And I'm running in my batch program as process ID 2. This is much handier. An example of this. This is pretty cool. Here, let me stop this again. Are we back on grumpy? There we go. Bring this back up. And at this time, I'm going to add on an extra command, or an extra option. Now I'm going to say slash bin slash bin slash PS dash EF. So what's going to happen to this case? First of all, I'm going to log in as process ID number 2. That one, that dash A does that. And because I have a program at the end of the line, I'm actually going to run that program. I'm going to run PS command. PS minus EF. Oh, let's switch it up. Let's switch it up. PSAUX, there you go. And now you can see that, yes, the stubbinit was process ID 1, just as we expected. Good. We ran our command, the bin PSAUX command, as process ID number 2. It executed the program and then terminated. Cool. That's kind of like a one-shot container, right? Run the program once when the program's over. Stop. So you can have all kinds of interesting things go on with that. Of course, if we look at our, oh, we can't look at the stack. Now, as I say, we look at our network stack, but there's no point looking at the network stack right now, because it's going to be the same. And we want to get back. We're ungrumpy already. So I think that what we might want to do is I think that we should go over and we should have a look at the network. So let's get our network going. So this is all our stuff. I'll take this out. And then we're going to add back in a network dash bridge is equal to verber 0. Run KVM on this machine so it has live vert. And live vert looks after the networking for KVM. So it just happens that we know what the name of the bridge is, so we can just use verber 0 on the way we go. There are other ones we could use. I have virtual bridge 1, 2, and 3, actually. You can also create your own. That's not a problem. So the flexibility and the networking is quite extensive. I selected a bridge just because that's the default configuration that matches the rest of the virtual machines on my system. So let's start it up with this bridge. I'm not going to put an A in at this time. I don't want to be just 2. I actually want to boot it like a real machine. So B for boot, got it. M is the WN3, so that's my machine name. And then, by the way, I'm going to add in a network bridge and connect it up please to virtual bridge 0. Let's run this up. This looks a lot better, right? We actually got what looks like a system initialization happening. So have a look, let's go back. Let's scroll back up the list and see if there's any fails. You might get the odd fail depending on where you got the image from. Like if you had Cloudinit as a package on your machine, the Cloudinit package may fail. But that's OK. It won't stop it from coming up. It just would take it out when I'm using it. I do have a warning. I do have a warning. It says the system getty slice unit configures an IP firewall. But the local system, that would be grumpy, does not support BPFC group and firewalling. Oh no, it does not. Whoops. It's a warning. So it's not going to work. We might want to change it, OK? Excellent. We could use some other firewall if we wish to, but not EBPF or BPF firewall. Cool.  There's my journal socket. That's cool. Using huge pages, that's neat. All right, login is rude. Password for rude. Got that. We're good. OK, so we're on grumpy. We're on 418. OK, this is not grumpy. Oh, yeah. Uname dash R. And we're actually our kernel that we have for our machine is 418-147.8.1. Red Hat Enterprise 8 underscore 1. So this is actually running the CentOS kernel, which is fine. We can see it. And that's what we expect. How's our network look? Well, let's look. It looks a little barren. So I've got my network adapter here. I've got loopback. Well, that's OK. And then I've got another adapter. I've got a host 0 adapter. This is the one that got created for the bridge. But there's not a whole lot of action happening with that. So I think what I want to do is I want to check one of the extra packages that we could use or extra facilities that we could use. And that is SystemDNetworkD. So let's get its status. And its status is SystemCTL. Oh, it didn't say status. Sorry. There we go. So it's here. It's disabled. The preset was enabled, but it stopped. And it looks like it's pretty dead. Why don't we start them up? SystemCTL, start. There you go. There's the verb. SystemDNetworkD. That'll start it up. Let's just make sure it's enabled. I know it says it's enabled, but I make sure that the links are there. Make sure that it's OK. So did it work? Yep. I got my address. So there's a 169 multicast address hanging around there. There's <IP_ADDRESS>. And that's going to match within my environment because the 192.168.122 is the Livevert Network. Now, depending on whether or not I've got DNS services, I might have to start up SystemDNamedD. ResolvedD. SystemDresolvedD. There we go. I knew it was there. To get your hostname resolution. Let's try it. Can we talk to the outside world? Can I talk to the world? Oh, look at that. I can talk to the world. And it's working. So I have a currently, I have a very handy little U-1-2 machine running on top of my sent to us machine. Why don't we go have a look at that other machine for a minute? And it should be right here. Did it work? Oh, yeah, it's there. So we've done the machine control on this thing, right? So we should be ready to go. I've got a sent to us eight dot raw. So we should be able to say N spawn SystemDN spawn. Let's go for a user. I don't know. We'll just go for nothing. We need to specify the image that we're going to be using. Sent to us raw. That's a good thing. Do you want to network on this one? No, this is its first day. Let's just bring it up. Who said we had to learn how to spell SystemDN spawn? No, no D. Oh, darn. There we go. We're running. Perfect. Cold sent to us eight. That's handy. So password, right? First thing we need to do. Authentication update is successful, even though there was some complaining. PS minus CF, not much going on here, right? Just as we expected. The network, we should probably see all the adapters. Yeah, we can see them. Actually, we can even see the virtual adapter for W3K. It's there. Eight was assigned to the device. Eight is assigned to the container, where it talks to nine on the other one. So that's the eight to nine connection. OK, so that works. Let's bring it up under boot mode.  Why don't we just throw everything at it? So bring it up in boot mode. And let's bring up a bridge at the same time. Network dash. And there are other networking options, not just the bridge. Today, here we go. Look at all that exciting stuff that this guy gives us. Woo-hoo. Oh, look, there's some warning stuff. Errors on network is unreachable. Failure failure. Yeah, that's OK. Let's let it come up. And then we'll go back and examine it and find out what it wants. Looks like network manager's having suits. It looks like it's looking for something, or other. I thought we said virtual, but sure, we'll find it. Well, it's waiting. Let's scroll backwards. Seal info. Cloud init info. So this is stuff from Cloudinit. I'm suspecting that Cloudinit is causing us some grief. And it's running failed security auditing service. So audit-deservice didn't start up. What else didn't start up? Remount root file system. We'll see how that turns us out. So you can see there's a couple of errors. OK. Think that we should be happiness yet? Not happiness yet. OK. So darn. Let's go back here. Let's bring it up without stuff again. Bring it up with very plain stuff. There we go. Our container's back up. So that's where it is. So let's say rpm-qa pipe. And yes, Cloudinit is there. We don't really want Cloudinit. So let's make it go away. So we're just going to say rpm-e Cloudinit morning. There we go. No such file. No such directory. Is it still gone? It's gone. Let's try and bring our guy back up again. Got to come out of here. Then we'll bring it back up and see if we have any more. Looks like Network Manager may be having a little bit of a there we go, a challenge. But Cloudinit was the thing that was causing us a problem. We did get one minor error up here a little bit. Failed. We're still failing on auto-deservice for our container. I don't know that I would worry about it unless I was doing some tracking and tracing. Now, can I log in as Uncle Root? Yes, I can.  PS minus EF. Oh, this will be the biggest one we've seen so far. Look at all the stuff that's running. So we've got SystemD and JournalD. And D-Bus is happy in single signs and single levels. SSD is running. GSS Proxy, Network Manager. Wow, look at all our syslog. Wow.  Can I run other programs and stuff on here? Sure. Why not? I could say DNF install, I don't know, but the NewLake Commander. Just to say something to install. Yeah, it's OK. It's a lot of stuff. But that's all right, we don't mind. And just to see what's going on, see if we can make it bigger. Goodie, goodie, goodie. Now we can go in and we have our blue screen of text. And we can go look at stuff and wander around, have some fun. Yeah, we can do all of those things anytime we want. I think that I'm going to go here and using the mouse to turn it off and that way it's gone. OK, so we've done a raw image. We had it converted. We did it from, we did an install through Deboot.  Actually, we have all kinds of cool things to do here. Now, we've started them. We've seen a couple of images. Let's go have a look at the making it N-Spawn service. Because I've got two different machines, right? Now, a local host of my ad here doesn't say Grumpy, so it's not me. So it's lost its name. There we go. Somewhere I lost this machine name. My good buddy, network manager, may have decided that he wanted to change it on us. Running systemd is a service. Yeah, there is a service already. And the service already is systemd-n-spawn. And then we pass it a parameter with the at sign. I'm going to say atDebianTree.service. And that'll start the service up for DebianTree. So it's a generic launcher that we can pass the parameter of the name that lives in machines. OK, now, speaking of machines, here's my two machines. What if I say machine-ctl-list? And it says, well, a lot much happening here. I've got a DebianTree running on one window. Where's my other one? It's not running, so I better go look. So I've got the other one I've got running as an image. So we're not going to be able to see it. And the<PHONE_NUMBER> is actually the host machine. So what we're going to do is that good old DebianTree person in there is that we're going to launch it through system-ctl. So why don't we stop it? Actually, did we look at the machine stuff? Machine control. Here's the list of stuff that it can do. It's a pretty big list. Cancel transfers, check the image status, list the images, list the transfers, pull a tar file and a raw file, and export and import, raw files, tar files. So you can get a tar directory of the root directory and just use that. That'll work as well. There's a terminate program and a status program. So you can see what it is. There's an S-top program. Well, I've got to run that. Here we go. Too few arguments. Oh, no. End demand pages, of course, are awesome. What I want to do here is just to stop DebianTree. Hopefully, it's stopped. Does it stop? All right. I'm going to find the machine because it's gone, which is good. Machine call list. No machines running. And no machines listing that in there. I got list of images, though.  The other thing was kind of nice. You saw the machine list, which was showing us what the IP address was. So that might come in a little bit handy. OK. Getting this to run. Now, when we launch from system CTO, we're not going to be able to use our big, long command line. Right? The big, long command line is going to be someplace else. All right. So what we're going to do is that we're going to use, well, we're going to use the other one, but we're also going to use an optional configuration file. OK. The configuration file is going to be under ETC System D N Spawn, OK, is where it's going to go. So we need to go find ETC System D N Spawn directory. Let's use the other window because we're going to use, not that. Where's all our options? We'll find them. So change to ETC System D N Spawn. Not here. The directory's not there. That's not unexpected. It's not unexpected. So you have to go make the directory. Then we can go change into it. And there's nothing in it. Surprised we are not. But we're going to create ourselves a file, OK? So we're going to go create with our handy-andy editor a new file. The file is going to be called debian-tree.n-spawn. Excellent. We have a user-friendly screen. Now, what we could do is customize it. So I'm going to use the exact section. And I'm going to say boot equals on. I'm going to say hostname equals lake, just so we show it, just a different name. Hostname's going to be lake, OK? I could tell it to use process number two. And there is actually a whole list of these things. If you have a look at the band page for System D N Spawn, there's a whole big long list. So that's all I'm going to exact. But I am going to put the network up, because I like networks. So here's my network section. I have a bridge, and that's going to be called serial.  So now that we've done that and saved it, instead of starting debian-tree with all those extra options, all we need to do is say systemd-n-spawn. And then we can say systemd-n-spawn. And I don't think I need any other parameters. Oops, I do debian-tree, brain. It doesn't look like a tree. Let's do it the way we're supposed to do it. Let's use systemctl. But we'll make sure metaphors. Systemctl, start, systemd-n-spawn.d, debian-tree.service. DebianTree service not found. Oh, I typed in the wrong. There we go. Only one little typo. OK, so I made a few typos. So my service should be running. Let's see. If I was to run machinectl, I would want to say let's just do list. It lists the machine out. So there's my debian-tree machine. It's a container. It's come off of systemd-n-spawn. It didn't do the OS version, but that's OK. But it's also showing me what the IP address is. And if I'm really lucky, we should be able to ping that, 22.88. We can talk to it. Since the network works, it's now a case of configuring other components and things that we want to use in order for us to use n-spawn. So this is a lightweight service. It doesn't take up a lot of resources. If we had more windows, we could open up more things and see that it really isn't using a lot of process and top power. It's pretty light. It's not that bad to configure once you get your image files there. It's got clone and copy features in it, too. So you can clone and copy your containers back and forth, import them, export them, and have all kinds of cool things with them. Do they migrate? No. They don't migrate from one machine to another machine. These are intended to be lightweight, simple containers. So they're only going to sit on this particular system. So if you're in an environment where you would like to do some testing, this is an awesome way you can do some testing, especially for administrators or developers or anybody that wants to have a nice, clean version of the system so they can have a little run at it. Pretty much concludes all of the cool things that we don't want to talk about this week. We do have an opportunity for questions and answers. And we can do that immediately after this, which is awesome. I would like to thank you very much for coming to see the Linux Foundation conference, our virtual conference of that. So thank you very much for coming and attending. And I hope everybody stays safe. And hopefully we'll see you again. Questions and answers are after the session. So thank you very much. I'll be there live now. I'm back. I'm going to check with our technical support people. You can hear the audio. Just double checking, that's all. Oh, no. All right. Sorry. Again, thank you very much for sitting through the session. The theory and the lab exercise, we had to make a little bit of changes due to our COVID environment. But we have all the information for you to go ahead and do that again. And I'll post it up on the sked. That's over by where the big schedule is. You just click on the session. And at the top, there's files. You can download that, OK? So everybody will have an opportunity to have a look at that. I've been trying to keep up with questions that have arisen. If you have any questions, I'd be happy to field them and see what we can talk about. I like it. I like EnSpawn. I use it for all kinds of little toys and tools. And I use it a lot for testing when I want to try out new environments, change operating systems in a hurry. So I use it a bit, quite a bit. And just in a very plain state, to make sure something like PostFix works the same on Debian as it does over on CentOS, to make sure that things that I'm creating and other environments are correct. So that's what I use it for. Now, questions. I've posted all the questions that I possibly could. I see a few people have wondered, which is awesome. But if you do have any questions, please post them down and I'll see what I can do about answering them. If you're looking, by the way, if you're looking at EnSpawn, it does know about C groups. So you can do a lot of work with C groups with EnSpawn. It also knows about, and my brain has just failed, the newer security mechanism that breaks down root into various sections. And yeah, it works really well with that. Now, I'm going to think of the name in a moment. It's going to take me a minute to figure it out. But it'll come capabilities. See, there you go. It was just a slow return on the packet there, sorry. But it does know about capabilities as well. So you can put different privileges in the EnSpawn containers. A couple of people have asked, would you use this over that? And I use containers, I use Linux containers, and I use EnSpawn because most of what I'm doing is not huge. I'm not doing industrial grade at this point. I'm using it for development, for experimenting, for testing, for performance, for compatibility, those kinds of things that I use it for the most. So would I put a database on it? Maybe not. That may not be the idea. Would I put an operating system up so that I could test what happens if I did this? Absolutely. Absolutely, I would do that. So those are the kind of use cases that I use it for. Kido. So check our queue for questions and see that they all pretty much have an answer or they've all been made public. So you should be able to see all the answers to the questions, which is awesome. And if there are no questions, then I'm going to say thank you once again for attending our conference. It's been a new experience for everybody, so it's kind of neat. And hopefully we will see everybody in person next time. If there's no other questions, then I'm going to say thank you very much and we'll see you again.