Hello, hello. So I'm Stefan Graver. I've got Christian Brunner. We both work at Canonical on Lexi, Lexi, and LexiFS. And today we're going to be going over how to set up a home cluster, something that you could use for a home lab or for a small team lab environment. Let's get into that. Yes. So most of us probably know. So people who are familiar with Lexi and with Lexi will probably know us from for our container work. This is what we are most experienced in. But we have ventured into newer areas with Lexi that we will talk a bit about this as well. So I guess most people nowadays are kind of acquainted with containers, but it might help to give a quick recap. Containers are, well, actually, there is no container concept on Linux. This is always the famous dictum that containers are a user space fiction. So that just means that the Linux kernel isn't actually aware of, doesn't have an internal container object. You can't just perform a system call and suddenly you're in a container. And this is very different from other systems. Instead, containers are made up of very different kernel interfaces. Most of them have to do with resource isolation or with security. The most famous one are usually namespaces. We have eight namespaces by now on Linux. The last edition was the time namespace. And they isolate certain aspects of the systems, such as the mount table or UIDs and GIDs. This would be the user namespace, the pit namespace. And then we also have C groups, which concern themselves with resource isolation. And there are various other interfaces that become important when you're trying to build a full working container. And for us, the goal has always been to create something which is a system container, which is in contrast to application containers. So application containers are usually containers that run a single binary inside, a single process. So they don't boot a full system. And in contrast to that, a system container is a container that runs a full, unmodified Linux system. It has always been our goal. So we make use of every available kernel interface that provides additional security to containers. We develop quite a bunch of these features to the upstream kernel. We integrate them into our products.  So system containers are in contrast to application containers. The oldest types of containers, as you can see. Also noted this on the slides. They have a long history, actually, container. So Linux hasn't spearheaded this concept. They have existed in the form of BSD jails, where they are actually a, and Stefan can correct me if I'm wrong, where they are actually a property of the kernel. So there is an internal object, internal container object, the same for Zularis zones. There was an implementation of something similar in the form of the Linux v-servers. There was open vset. And then around 2008, Lexi started. And then around 2014, 2015, Lexi was started. And in between around 2013, the whole application container thing that we know nowadays has become more and more associated with what a container actually is. So but our containers behave like a standalone system. So we don't need any special software. We don't need any special images or root events. We can basically, if you have a tool to create an image or root of s from your distro, then you can use this. And Lexi, in this case, will exactly know what to do with this, and you can just start the container. The advantages that this is really low overhead, they're very easy to manage. This is something to do with how we designed the API, which Safan will be talking about later on.  It's very transparent. So you see all of the processes that run in the container appear in the process tree of your system in contrast to, for example, virtual machine. But as I promised, we're also ventured into new territory. While the goal is obviously or has always been with system containers, that they cover every use case that virtual machines that you normally would use virtual machines for, that isn't necessarily always the case. There are still some limitations, like, for example, we can't fake or virtualize hardware. And that is why virtual machines became a natural extension. And we have already had experience in this area. So this is why Lexline is also able to manage virtual machines. Well, there's obviously a very obvious glaring difference. And that is that containers and containers share the same kernel with the host. That is, for example, also a limitation. So if you want to run a newer or older kernel, you can't do this with containers. You would need to update your host. They also have virtualized hardware in firmware. And they behave much more like a physical system in a lot more detail than a container or system container would. They are hardware accelerated due to how the Linux kernel works and the KVM module in the kernel. And useful virtualization actually requires hardware support.  For example, from using virtual utilization or where devices, bird IO devices, these are usually called. And this is a trend also in the upstream kernel. So usually when a piece of virtualized hardware becomes important enough, then there will also be a bird IO driver for this. You also have new file systems developed that you can use with virtual machines that try to make up for some of the limitations that you have with virtual machines in terms of IO performance and sharing directories and folders with the host, which is very easy to do with containers and which is obviously, for obvious reasons, very difficult with virtual machines. And another advantage is that it can just run about any operating system. So with system containers, you are obviously constrained to running Linux distributions. You can, for example, run, I don't know, Mac OS or Windows in virtual machines. So if you have use cases or workloads where you require Windows, then virtual machines are obviously the way to go. And newer versions of LexD will actually support this. OK, so going over what LexD itself is, Christian already did a bit of an overview of that. But LexD is effectively a container, system container, and virtual machine manager. We don't directly care about application containers. You can run your Docker containers nested inside LexD, either inside the virtual machine or inside a LexD system container. Both work just fine. But LexD's focus is really on giving you like a way of managing effectively machines like instances that run full operating systems. It's really quite simple. We've got a key incoming line interface that drives a REST API. So it's easy for external tooling to integrate with LexD. It acts in much the same way as a local cloud on your machine. It's image-based, so you don't have to go through either manually assembling the root file system of a container or manually installing a virtual machine. We've got pre-made images. And that makes things much faster, much more reproducible. It's very fast. It's got optimized storage and a migration API. It's got support for a variety of direct hardware pass-through both into containers and virtual machines. It's also designed to be secure. So we did a lot of kernel work around the user namespace. And we were definitely early adopters of that in LXC. But it was quite difficult to make use of it. And more importantly, not to the default. With LexD, containers are unprivileged by default. The user namespace is used by default. And we do use things like SecComp and Aparma by default on all of our instances. You can go and tweak things. You can always make them less secure. You can also make them more secure at the cost of some features. But we do have a sense that we want things to be safe by default. It's quite scalable. You can go from running LexD just on your laptop desktop Chromebook, and then move from that local experience to running on a few servers that we'll be showing you today with some random development boards. Or you can deploy it on large clusters, like rack scale, 40, 50 machines. That can run tens of thousands of instances. The API will be the exact same one. Something that you write that can interact against your laptop will be able to interact against a massive cluster just the same. There are additional fields that become available as you end up in an environment that cross multiple systems. But if you don't understand those fields, it's perfectly fine. LexD will just try to balance things for you and take away any of those new features. LexD itself is written in Go. It uses LibLexE to drive the kernel container bits. And it uses QMU for the virtual machine layer. As I said, it exposes the rest API for its clients. And we've got a number of those. We've got our own CLI tool, which is what we'll be showing in the tutorial part later.  And here on the inside, you can see how things might look like in LexD deployment with multiple hosts where they all run Linux. They then run LXC or QMU, or they can use both. You can totally run virtual machines and containers on the same system.  It exposes the rest API, and then we've got clients talking to that. What's not shown in that diagram is what happens with a cluster where the bottom layer would pretty much merge together inside a single API endpoint that would be combining all your machines into just one instead of having their own machine view. As far as the components of LexD, let's go through the main ones. Obviously, LexD runs instances, so what you see in the middle. Instances can have snapshots and backups, so that's additional endpoints for that. They are created from images. Images can have aliases, because we reference our images based on their shot 56, and it's not really pleasant to type those, so we've got nice names that you can use on top of them. We've got networks and storage, because you usually need those two bits for a working instance. So you can create a variety of different type of networks through LexD directly, and for storage, we also support a variety of storage drivers. Things like barfs, ZFS, LVM, serfs, FFS, and plain old directory of things we support. You can create storage volumes on top of those storage pools. Those volumes can themselves have snapshots, and those volumes can be attached to instances with other containers or virtual machines, and you can then easily do a shared storage between them that way. There's also the cluster layer at the top here, which lets you scale the cluster, but also list its state and see that everything is working as expected. And somewhat recently, we introduced the concept of projects, which lets you segment your LexD into multiple chunks, effectively, that each can have then the same instance names, but not see each other's instances. That's quite convenient when you've got multiple software that might be deploying LexD instances, and your rates that they might do configuration changes that impact each other, or that they might just come up with the same instance name and similarly conflict. So that feature lets you separate things. In a shared environment, it can also be used to give each user of a shared cluster, for example, its own view of the entire thing. Then we've got a few internal endpoints, things like certificates, that's to manage access to LexD over the rest API, because we use TLS for that. We've got an endpoint that gets you live events of what's going on in LexD. We've got another endpoint that shows you all the background operations that LexD is running. And then we've got a few more APIs that are not even listed there, but sits on top of the instances for things like file transfer, executing commands, attaching to the live console of the instance, or even publishing an instance or an instance snapshot into a new image that you can distribute. Yep. So a lot of the question that you might have is, where is LexD actually used? Well, actually, LexD is used in quite a few interesting workloads, which is going to highlight two a little bit. So currently, on most recent Chromebooks, the new Linux subsystem that they've implemented is based on LexD. So it uses LexD internally. They basically implemented a client tool for LexD, but they also, with some trickery, let you give you access to a LexD client tool itself. So you can actually not just run the mend, well, mendated Linux distribution or container that Google wants you to run. You can run any container on a Chromebook, any container that LexD supports on a Chrome. So this is pretty cool. Any new Chromebook will be rolled out with this feature. You can enable it, and then it's powered purely by LexD. It runs on privileged containers by default. As we've said, this is a sense that we're taking. Everything should be super secure. And this is Chromebooks designed for exactly that purpose, being as secure as possible. So this fits nicely into the story. And it relies on a lot of the things that Stefan just mentioned before, snapshots. It uses our backup API, which we have. It uses File Transfer and USB GPU and Generic Unix device password that Stefan will probably mention later on. The File Transfer API is especially cool because you can push and pull files from and into the container. It also makes use of our ability to expose and share directories between the host and the container. So any files that, for example, if you install an app on a Chromebook that runs on Linux, then, and you store files, it will go into a dedicated folder that you can access from the host. So it's tightly integrated with the whole Chromebook So it's pretty neat. It's pretty neat. That's pretty exciting work, actually.  So if you have any, if you test on these kinds of architectures, they will make use of Lexity. And the cool thing is that they make use of very advanced features in Lexity that we have, that we didn't even, that we don't have separate slides for, but they are pretty interesting in and of itself. So obviously, they're running containers. They're not using the virtual machine parts of Lexity. And because, obviously, these are architectures usually that are either very expensive or not as widely available, so they need to be, they need to make sure that they can run jobs as densely as possible. And for that, containers are just super nice, and both Lexity and Lexity build on all of these architecture. We've almost built on any architecture that you can think of, but especially for these, it's pretty cool. And since they're running unprivileged containers, but often users have use cases where they want to make use of more advanced features, like sometimes your workloads need to create device nodes, or your workloads need to mount something that you wouldn't be able to mount. So we've implemented a feature in the kernel that is called syscallinterception that allows you to intercept and emulate syscalls in user space, which is pretty powerful. So usually any make not syscall inside of a container would fail because that's just how username spaces work, and it's a security mechanism, so you cannot use this to attack the host. So otherwise you could create random device, multiple block devices, and character devices, and then suddenly you could write into random kernel memory, or you could get access to a device that you shouldn't have access to. But at least for some of the devices, Lexity can guarantee that it's safe to create and use them. For example, they have zero and dot null, we already bind mount them in from the host. So we already vouch for them to be safe, so there is no reason why workloads, for example, when you run the bootstrap or something else inside of container, why they shouldn't be able to create device nodes, these device nodes. So the second syscallinterception work basically blocks the process instead of the container that performs the syscall, and then we'll call out, the kernel will inform Lexity container manager that a syscall for a device has been made that we think is safe to do, and then Lexity will inspect the arguments and will verify that the container is doing something, the process that it's doing, the syscall is doing something sane, and if it's decided that this is okay, it will then create the device node for the container. And so the make not syscall inside of the container will succeed, and then it will instruct the kernel to report back success to the process that performed the syscall. This is something which we extended quite a bit. You can nowadays, we're thinking about making this even more powerful, so that, for example, that workloads like Travis or workloads that you have can even do more interesting things. We made it possible to inject bit descriptors into a task and to retrieve bit descriptors from a task, so you could technically intercept the open syscall and then open a file descriptor for a file or directory that the container would usually not be able to open and then inject the file that the Lexity Container Manager opened for the container into the process and then make the open succeed. So this is a pretty powerful mechanism that you can also use for rerouting, networking, and so on, and this is powering Travis, so this is pretty cool, so pretty excited about that. So on, and like some of the requirements around Travis was also to create those instances as quickly as possible and to have Docker be fully usable inside of the instance, so that's what drove a lot of that stuff. Like we have, for the under creation side, Travis is effectively using Lexity with ZFS on NVME drives so that you can, and like a pre, we've got a pre-created container gets copied on a per job basis that can take as little as a second effectively to create and start that container, so that's very, very quick. And then separately, a traditional EXT4 volume is attached for Docker to run against and that's where some of the syscall interception comes into play to make Docker work properly inside that environment. We also had to be quite strict on limits and restrictions to make sure that one workload can't affect another and that the workloads can't harm the host system and that it will go away once it's done. So it's quite good to see it used at this scale and with that kind of environment. Yep. And we promised you, so talking about this, the container use case now first, we promised you that you could run basically any Linux distribution inside of containers that you care about. And this is just a rather random and non-complete selection I think of the images that we provide. Maybe it's, is it all of them, Stefan? No, it's not. We have a few logos on there. We actually support building, we're going to build 20 different distributions so that we don't quite fit. And we, yeah, we build like 20 distros across 94 releases for containers, just like 390 images we build daily. And all of these images are based on if available are based on the official images that these distributions distribute. So if your distro has an official image then we will use this. If it doesn't, but it provides a way to generate these images in an official way by providing a process then we will do this instead. So there's nothing modified about these images. And yeah, as Stefan said, they're generated daily. And we have a huge build farm to support this. And you can see we have Alpine, Alt-Linux, OpenSusapadora, Ubuntu, Oracle Linux, even Debian, of course, Arch Linux, CentOS, Guento Linux and also Ubuntu Core. And this is, we are serving currently over a half a million images every month which is in this case downloads not launches of container images. So, and this is as far as I can tell this is just going up over time. So we're pretty excited about this because it shows that Lexi is seeing more and more adoption. And nowadays, since we started supporting virtual machines which we mentioned before, obviously we also build virtual machine images not as much as we, not as many I think as we do for containers, but still quite a lot. And so you can launch, you can launch virtual machines in the exact same way that you can launch containers which Stefan will then later on show in the demos which makes it a cool experience because there is no difference in experience between launching a container and between launching a VM. So there you have it. Might be worth mentioning that the image build process that we effectively do it for that is we first assemble the container image. So we pull from the, from the extreme destroy image or whatever artifact we can get our hands on. We turn that into a container with file system. We generate that image. And then as a second step we install camera and bootloader in that same image and we generate the virtual machine image from it. So the actual, like if you install for the same build the container and the virtual machine you'll know that all the packages are gonna be at the exact same version, the exact same location, the exact same thing because they are the same thing. We've literally just added a camera and bootloader from the distro before spitting out the VM image that's really the only difference. Yeah, so that's pretty neat. Right. So lexity system containers. We've touched a bit on this before but it's probably worth going over this in a little bit and a little more detail. So this is our, really this is our area of expertise. This is where we have the most experience in and I think it's safe to say we have the most featureful and safer system containers out there. And we're still eating in feature development in the kernel and also in user space to make as many workloads in our containers possible as we can do. I mean, this can range. So this can range from, for example we've written a tiny fuse file system that emulates certain proc files that show host values to show container specific values which is quite important to think for example about sorry about running a Java workload inside of a container and the JVM, the Java virtual machine will look at how many CPUs are available on your system. And let's say for example you dedicated a single CPU or two CPUs to the container so it can only run tasks on two specific CPUs and Java checks proc CPU info and then detects that your system actually has 256 threads and will then start to spawn 256 threads or even worse when it reads your system memory from proc and finds out that you have 64 gigabytes of RAM available but you've only given your container a gigabyte of RAM and then JVM will decide it wants to allocate eight gigabyte or 16 gigabyte of RAM then you're obviously going to run into problems. So this is why we wrote LexiFS it for example, it virtualizes virtualized CPU info and all of the memory information based on the C groups that the container is in. So we do a lot of this, we do a lot of this stuff that is more or less behind the scenes because it just enhances your experience of a system container, but it's actually quite crucial and is also quite critical and interesting work and something that we maintain as well. So this sort of relates to the comprehensive and flexible resource limits too. So we have CPU limits, which I talked about you can for example limit the number of cores that you're exposed to the container but you can also determine how much shares, CPU shares the container have. But this is, there's a very fine-grained API that Stefan developed for and designed for Lexi you can obviously also limit memory. I mentioned this in the JVM example too so that you can make sure that your container only has one gigabyte of memory available so that it can exhaust the memory limits of the host. We can have network limits, I think, in root, in ingress and egress limits, right? We can specify your MTU limit. We also have limits on storage, but this is obviously specific to the storage driver that is used. So if you're using ButterFest then you can use ButterFest quotas if you're using Setafest at your backend which Lexi supports, then we're using Setafest quotas and if you're using LVM we're using a different kind of quota, XFS quota. So, but in general, we make sure that you can also limit storage. Your storage consumption, I'm sorry. And we also have limits on the number of processes so that you can for example, say this container is only allowed to have run that many processes. Again, this is done through C groups and we also expose or let you configure various kernel limits. Like usually you don't really need to think about this too much because Lexi will choose same defaults. This is the general idea but you might have specific workloads that make it necessary that you have very fine grade control over how many memory you expose to the container, how many CPUs to expose to the container and so on. And so we have this all documented obviously and then you can look at all of the limits that we have. One of the interesting features, one of the interesting features is device pass through because this is really exciting, right? We have talked about, we have briefly touched upon the fact that virtual machines are limited in the sense that they need to virtualize a lot of devices which obviously makes them slow. And I mean, you have PCI pass through and so on and that's obviously a nice solution but with containers this is way easier. One of the obvious examples is you want to expose a GPU to the container to for example run an otherwise isolated computation intensive workload, tensorflow or what have you inside of the container and GPUs are just very nice for this. So we have exposed and you can expose GPUs directly to the container, we provide you with an API for that. We've also integrated or a bunch of friends at NVIDIA helped us to also make it possible to easily pass through NVIDIA GPUs including all of the CUDA libraries that you need to the container because there is a dependency between the driver that is used for the NVIDIA GPU and to user space libraries that are used to run your workloads. And if you for example, run an older distribution inside of the kernel and install the CUDA libraries then you might get into conflicts with the actual GPU driver that is used on the host. So we provide an API where the CUDA libraries from the host are made available to the container. We also let you pass through network devices. So for example, if you have a server with multiple network cards, physical network cards you could decide that this container is supposed to get its own physical network device and then you can pass it into the container. Now the separate, I guess the separate network device case is not the most common one to be honest but if you think about Infiniband devices where you have where you split a physical function into multiple virtual functions or into separate virtual network devices that are still very, very fast and you can have up to, I don't know, 200, 500 of those and then you can tell XT to take one of those devices and pass it into the container. We also expose all of the character devices that belong to this and Lexity will take care to correctly detect all this. So this is pretty advanced as well. And we also allow you to hot plug USB devices. This is pretty cool and a nice feature that I use a lot. So for example, when I need to do something on my phone I usually don't want to install on my Android phone in this example. I usually don't want to install all of the 80 Android development kits and so on on my host and then remember to uninstall them and clean up all the configuration files. Instead, I can just install them in a container and then tell XT to expose my phone based on the vendor ID or the product ID to the container and then I can just mess with my phone in the container. So this actually, this is really powerful and really flexible. We can also nowadays detect when you for example, tell XT you want your UB key to be automatically made available in the container automatically. We were able to detect this as well. So you can hot plug devices. So if you take it out of your computer, your UB key or your phone, then XT will remove the device and you plug it back in then Lexity will detect it and create device note again for you. Pretty powerful mechanism. I like it a lot. We also allow you to make disk devices available to the container, which in this case, you usually mean like for example, you want to mount directory inside of a container or some sort of dedicated disk that you created for the container to share data and storage between containers, between multiple containers. That's actually quite an interesting use case. So you can hot block additional storage volumes to a container and that's all done at runtime. I should say that like all of these things can be done while the container is running. So you never need to restart container to any of this stuff. And obviously we allow you to add arbitrary character and block devices. So you either give Lexity a path for an existing block device and Lexity will create it in the container for you or you give it the minor and major number of the character block device and Lexity will create it for you even though it might not even exist on the host. As long as you have the driver available in the kernel that should work fine. So this is pretty powerful and we allow you to hot block Unix character and block devices too. And then we have a bunch of advanced features, a very specific feature which we have briefly touched upon already is system call interception work that powers Travis. So there is no need to actually go into this with more detail. But we also care about, for example, we also take care if you hot block a USB device inside of container or a UB key or a network and device. Well, actually network devices are not a good example but USD devices or your UB key then what the kernel usually does it generates a U event. And this U event exists to indicate to, for example, UDEP which is a device management daemon that usually nowadays is shipped alongside system D to make it aware that a new device has been added to the system and that it can now start to initialize the device or run hooks or whatever, what have you. And these U events traditionally weren't namespaced. And so you would only see them in the initial username space meaning you would not see them in the container even though the device had just been created or hot plugged into the container which is why we did some work to actually inject U events into the container.  So if you have any device management software in your container waiting for this device to show up then it can get notifications via U event because LexD is taking care of you. So this is pretty nice. We also in terms of security allow you to generate isolated ID maps not generate isolated ID maps but running containers with isolated ID maps. What this means is each container has a non overlapping UID and GID mapping which provides additional security because containers can exhaust each other's R limits. Also if you break from one container into another container you cannot do anything. So this is a pretty powerful security feature that users which care about security obviously make use of a lot and it's also used on Travis again. And then we've worked on something called ShiftFS which we currently only carry as a Ubuntu specific kernel patch but the intention is to make this available at some point in the upstream kernel as well. And this for example, think of the case where you have containers with isolated ID mappings and you want to be able to expose a disk device from the container from the host to all of these containers they all could none of them could write to the actual disk because their ID mappings would prevent them from doing so with ShiftFS it's basically a translation layer that allows you to enable these containers to actually write to disk. So this is what we do with the with ShiftFS and we are working on an upstream solution right there. And I think this should cover more sort of a little deep dive into how system containers work. And Sophan will now go on to inform you a little bit more about our virtual machine implementation. And so on the VM side of things it's still quite new to us. We've only been doing that work for a bit of a year now, which is quite a good thing because as the new kid in the block pretty much as far as virtual machines goes we get to build on all the new hardware and kernel features that have been landing over the past decade pretty much. And we also, because we don't have to care about legacy workloads so much and we control most of the images running on those we got to pick a lot of newer features and really build that story based on that. So Lexi runs modern virtual machines. That means the only firmware we support on Intel and ARM is UEFI. So we don't do legacy by us at all. We've got security enabled by default. We do use virtual devices whenever possible. And right now that means all the devices we have are virtual devices. They're not faked Intel or faked NEC or whatever else devices that they're only using the modern Bataio devices. We run on QMU 4.2 or higher. I think we're probably gonna be bumping that to QMU 5 or higher, given what we're using these days. As far as the user is concerned the handling of virtual machines is as we mentioned earlier, near identical to handling containers. You, the only difference is that it's typed to virtual machine pretty much. And so long as an image exists, it just works. There's like no particular VM knowledge that's required for many of our existing API clients. They can just by setting the flag virtual machine they get a VM but then the rest of the APIs for things like executing commands, transferring files all that stuff works the exact same way which obviously is a bit of a challenge with virtual machines because unlike containers you don't get to just go and like spawn processes and mess with their file system as you want. But LexD has a agent that you can run inside the virtual machine and which is bundled with all of our images that agent communicates with LexD on the host and effectively lets the host system forward API endpoints that could only be implemented inside of the virtual machine. So that's how we do execs, how we do file transfers that's how we do putting detailed information about number of processes running inside the system all that kind of stuff. And all of that integrates seamlessly with our existing features like network storage projects, profiles. They're just a different instance type as far as we're concerned but pretty much all of our existing limits and restrictions and options apply the exact same way. Most of the device types also apply to virtual machines. The main difference really is that you don't get away with doing like unlimited type limits like a virtual machine must have a set number of CPU and memory, you can't just have it be whatever is free on the host which is kind of what containers do by default. So for virtual machines with the false to one CPU one gig of RAM and then you can bump that to whatever you want through limits. The other difference is that unlike a container where we can very easily reshape it and reconfigure it it's a bit harder with virtual machines. So we're slowly adding live update options but still the bulk of devices and configuration options and virtual machines right now we require a reboot of the VM that is. We do support like a bunch of the advanced stuff that Christian covered in containers. So things like SIOV for infinite networking we supported for infinite and infinite we supported just fine with virtual machines and that means we do VFIO effectively to pass PCI devices straight into virtual machines. We also support VFIO pass through of GPUs. So if you've got dedicated GPUs you want to pass virtual machines, you can do that. We currently don't do SIOV or MDEV on top of GPUs but that's coming soon which will then let you slice existing GPUs and pass slices into virtual machines. That being said, don't have your hopes up there like no, pretty much no consumer grade card can do it. Some Intel cards are capable of doing basic MDEV but if you're looking at something more powerful like an Nvidia GPU you're gonna need at least Quadro if not a Tesla type card to be able to get shared GPU with multiple virtual machines. So for those cases, you're probably a lot better off with containers because with those you can literally get away with sharing a normal consumer grade GPU with multiple instances. All right, and now just on to clustering. I promise we're pretty much done with the speaking, we're gonna go to the actual demo into your part next. So clustering is something we've had for quite a while now in next day probably coming up two and a half years or something or not. Effectively the way it works is that LexD is capable of talking to other LexD demons and has a built-in distributed database. So you can configure one LexD system then configure a second one and cluster them together at which point the API will show you the aggregate of both and if you join more and more nodes that just keeps growing. There's like no external dependency for this at all. It's all built into LexD and our default database is already ready to be clustered right out of the box. The API as far as the user is concerned is still the exact same one. It just feels like they're talking to a very large LexD as far as the number of things that are running on it. But if they're not aware that it's a cluster then things still work. The cluster will just pick whichever node has the least amount of instances and then spawn things there for you. If your client is aware of clustering then it can figure out where a given instance is. It can move instances between cluster nodes. It can request an instance is spawned on a particular node. There's a bunch more features are available if you're aware of clustering but if you're not it still works fine. And that it should scale to a very large size like LexD clusters are meant to be about rock scale. So up to 40, 50-ish nodes. But each of those is usually capable of running a couple of thousands containers. So you can quickly get to very, very large scales. And it's also where the work we've done around projects starts making a lot of sense because you don't really want your list of containers to include tens of thousands of entries. It's really, it's gonna take a long time to list that plus it's also not gonna be the most easy thing to read. And the likelihood of you like randomly getting conflicts or modifying a profile and impacting all your containers is pretty high. So that's where projects make a lot of sense. You can segment that cluster into some smaller chunks and then give individual projects to different software or different people. And they each gets to do their own thing, modify their own config and not impact each other. We also support mixed architectures so that you can have multiple different CPU architectures within the same cluster. And next day, we'll transparently deploy things on one or the other based on what's running the fewest, which is our default balancing method. Or if you ask for a particular architecture for a given image, it will obviously pick a suitable system for that. And you can also still target to force a given image to be run on specific machine of specific architecture. All right, so that was it for all the theory. We're gonna go on to something a lot more hands-on now. Nice. Let me nearly switch over that. I'm just gonna keep the slide up. And the first thing I'll do is, can describe what we have. So we're gonna be showing a demo of how to set up a home cluster initially with three Raspberry Pis. So there are Raspberry Pi 4 8 gigs each in this case, but you could use the one with less memory, especially if you don't care about virtual machines. I probably wouldn't recommend using the one gig one, but like four 8 gigs are really good ones with two gigs being probably good enough. So we've got those three Raspberry Pis also two Intel NUCs, which are Intel, obviously, based CPUs. And we're gonna be trying to cluster those together, run some containers, virtual machines, do some storage stuff, maybe play with stuff a bit and just see how that can all behave. So first thing I'll do is I'll just join back from my phone and just show you what the hardware looks like downstairs. Be back. This is going to be interesting. Stefan usually has a lot of interesting hardware at home. Okay, so I should be back from my phone now. Just gonna be heading downstairs real quick. I'm gonna show you what the stuff looks like down there. I can hear you very well. And so we'll probably do the others who are watching these videos. Well, it's about to get a bit noisy. Basement is always a bit noisy, sorry about that. Let me just flip around. Oops, there's my hand in front of this. Let me just flip around the webcam, the camera on the phone. There we go. Okay, so as we're looking out here, that's Raspberry Pi, one Raspberry Pi, two Raspberry Pi, three. They're all in plastic enclosures with a small fan, which is like one of the standard Raspberry Pi for kids. And then next to them, we've got the two Intel Nux. So let me just head back up slowly, but there we go. I can just say that both of them have Ubuntu 20.04 server basic images installed on them. I did that ahead of time because preparing SD cards is not really the most exciting thing to show in a talk. And I've got the SSH set up on them so that I can easily show all of them going. Close the phone now. And I'm back. So let's switch over to that terminal. Here we go. And here we can see that we have a terminal connected to our Pi 010203 and Nux 0102. We'll start with the first ones of the Raspberry Pi. Just move stuff around on my screen a bit here. And so out of the box on Ubuntu 20.04 server, LexD is already installed, but it's installed using our LTS branch. So that's what you can see here for three. And for this demo, we're gonna want to do a couple of things. So first off, we're gonna want to enable ShiftFS just to make things a bit smoother like a little on by avoiding needlessly shifting the file system. And then we'll refresh to Latest. So that's gonna move us to LexD 4.6. We have a long history of providing our users with a stable branch because a lot of our users actually care about staying on a stable branch that doesn't see a lot of the feature development that we do. We only provide bug fixes. So since we had already taken with LexD and we're doing this with LexD as well, but if you want to get all of the cool stuff then switching to, in this case for the SNAP, switching for the Latest channel is usually a good idea. Okay, so I've quickly done it on all the others too. So I went on 4.6 on this one. No, it would work better if I hit Enter on the second comment on those two. Let's do that. Okay, so while we do that, let's just do the initial setup on the first one. So we're gonna be running LexD in it. And for anyone who's done that before, you might have seen that the first question is, do you want to do LexD clustering? And usually you answer no to that one, but today we're gonna answer yes. It asks for the name of the node, which is R5-01, it's fine, IP address of it. So I'm just gonna make a quick note off so that it's easier to set up the next ones. Okay, and ask whether we're running an existing cluster which we're not, it's a new one. So we can set up the password, let's do that. And now the clustering piece is effectively done for that machine. Now we're just moving on to configuring other things like storage in this case. We're gonna go with the default of just doing as the first pull. We don't have a dedicated drive which is a bit unfortunate because those Raspberry Pi's effectively just run off their SD card and that is quite slow, but I don't have three external drives to attach to them to make them faster. So we're just gonna be using built-in storage and we'll make it 20 gigs large. There we go, no remote storage at this point. And we're gonna be using a virtual network that just comes with FlexD and on Ubuntu, which is the Ubuntu fan. And that one does need to know what my subnet here is. There we go, so just enter that one. Most often the automatic mode works for that one, but in this case because the NUCs are actually on a different subnet and needs to give it the entire thing, otherwise they're not gonna be able to communicate. So just fix that and give it a few seconds and we're done. So our Pi is zero one. So the first Raspberry Pi is up and running. We can look and we're gonna say that it doesn't run anything yet, which is good. It's not supposed to. And we can see it in the cluster with nothing else. Now let's go on to Raspberry Pi two. And you'll see that things go a bit faster as you have more nodes because it doesn't ask you all those questions again. Most of the questions are cluster-wide, so they don't, doesn't make any sense to ask them again. Kim, so name and IP is correct. We're joining an existing cluster this time around. So just gonna be entering the IP of the first one we did. It shows you the fingerprint, which you could go and validate if you're worried that your traffic might be intercepted. It asks for confirmation that there's nothing going on on the system because Lexity will be wiped, which is fine. It asks for the size of the pool we want, just in case it needs to vary between systems. The old identicals will go with 20 gigs too. And there is, the source would be a block device if we had one. Again, don't have any on those Raspberry Pi. So I'm just gonna go with empty pool name. Also we don't care. Just gonna go forward. You did mention, you're probably worth mentioning that the fund didn't do it already that if you are on, for example, on a single system and you just wanna initialize Lexity that we also provide an automatic switch that does basically choose the same defaults for you. This is really an advanced configuration of Lexity. And we also support initializing Lexity from a predefined config file in YAML format. Yeah, we can show the YAML in the next one to show the output. But yeah, the last questions of an interactive Lexity in it is do you want the YAML, which you can then use to just preceded on other systems. So now we can see that second node was done. And on that one, if I do Lexity cluster list, now we see that we've got two nodes on the cluster. You're also not gonna notice that the database is only active on the first one. It's because the database is a distributed database based on raft. And for raft to work, you need a column and you can't have a column when you have two systems. So instead of putting the database on both of them and then having the entire thing fall apart if either of them dies, instead we're keeping the database running on just the first node until you join a third one. So that's what we'll see now if we go on that one. That's the third Raspberry Pi. Okay, so would you like to use clustering? Yes, IP is fine. We're joining the cluster, the IP address, fingerprint hasn't changed last time around. Can wipe the entire system, it's fine. No source, no ZFS poll name, and the size is 20 gigs. And this time let's print the YAML, so you can do yes. And what it shows you is the YAML you could use to then reproduce that setup. So it shows the cluster details, the answers to those questions I gave, the cluster certificates, who the target is, and what the password is that I entered. Good thing I didn't put anything after you confidential in that password. All right, so now we've got, let's just clear the screen. We've got a cluster of three and we're gonna see the database is now replicated. So all three have it, which means we can reboot only one of the three and still have a perfectly functional database. With that done, we'll do the next thing a bit for now. Let's just do the most basic launching in the Ubuntu 20.04 container and call it C1. So that's just gonna be done loading. That's that's effectively picked a node for us, most likely the first one. So I expect it to actually run on RPY01 even though we spawned it from zero three. That's told it to the node the image. The image is then being unpacked into a new ZFS dataset and then the container is created and started from it. So if we just so now, the interesting thing is that it gets faster as you do it again. So probably not in this case because of the, well, actually I can make it fast. I'll do that after, but we can show that it's running. It's got an IP and it's running on RPY01. And we can totally get a terminal inside it from RPY03 because everything is clustered and we just the lexd API is constant and falls internally as needed. So that was just fine. Now, if I wanted to make it fast because we're by just creating a second, one of those containers we can do it with time this time. And I'm gonna actually set the limit because that's really interesting to show. So let's do two CPUs, two gigs of RAM. And if I was to just run it as it is now, it would pick another node because it wants to use the least used one. And that would mean spawning it on RPY02 which doesn't have the image. So RPY02 would then get the image from RPY01 and unpack it in a new dataset and then create which would be slightly faster but probably about as slow as we saw on the first one. But now if I tell it that I actually want it specifically on RPY01 that already has the image and already have the dataset, that should be pretty fast. Yeah, as we can see, three seconds later. As a reminder, this is not on a fast laptop or anything. This is on a small arm development ball. Like it's, this is on a Raspberry Pi. Yeah. So that is pretty fast. Now, if we can easily see what we have in C1. So in C1, I had the entirety of the memory. So eight gigs and we've got four CPUs. If we look at C2 that we did limit and we look at CPU info, well, and this is two CPUs. And if we look at the memory, we've got eight gigs, which is wrong. Why do we have that much memory? This is interesting. Anyway, we'll have to figure that one out. Let me make sure that the limit itself is applied and it's just, it could just be an issue with the rendering in CFS, so. Yep. Oh, not, huh, okay. So I did not know that, but apparently, so it's not like this fault. It's not the configuration's fault. It's a kernel limitation on the Raspberry Pi kernel looks like. So those, those both are running the one to Raspberry Pi kernel. And as you can see, there is no memory controller on that list. So that's the list of C-group controllers, but there is no memory controller, which means you can't actually limit the memory. Next, we'll just ignore the limit because there's no way to do it. Or interesting, could you go into the unified hierarchy? It should actually have showed in the, in the log on startup that it's gonna be. Yeah, we can see it towards the top of the screen. Couldn't find the C-group, both block IO weight, UTLV memory controller and memory swap controllers all couldn't be found on the system. And to answer Christian's question around the unified controller, no, I don't expect. No, there's only memory pressure. This is always there, yeah. Yeah, so that makes, that just explains why the limits didn't stick. We'll see it stick when we do the same thing on the next after, because on X86, we definitely have that in place. The other thing we can show is if we go into C2 and we look at the disk, we can see all 20 gigs are accessible. And we can totally change that by just overriding the grid disk and setting a size of two gigs at which point we've got the size of two gigs. Nice. That was perfectly fine. Again, if now we'll figure out X-List, we can see the two containers running and you see snapshot zero. You can create a new snapshot by just making a snapshot. This will have created a snapshot called snap zero. Yeah, we can see it down there, but you can also set up a schedule for your snapshot like so. So that's like a cron type pattern and that says that every minute we want a new snapshot of everything. So that's gonna be interesting to see over the next few minutes. Now to show different operating systems, well, different Linux distros, let's do Arch Linux. So I'm creating a third container called C3 that's using the Arch Linux image. Because we placed two containers on RPY-01, I'm definitely expecting this one to land on RPY-02 or zero three with my guest being RPY-02. As we will see that soon enough. Again, I would expect this to be quite a bit faster if you had like a USB 3.1 drive or something connected to the Raspberry Pi's. Me using a fastest D card is definitely not happening here. Okay, so I've got created and it's on RPY-02. It's still booting up because it doesn't have an IP yet, actually. Now let's do CentOS 8 for good measure. And that one I would expect to then land on RPY-03. Worth mentioning, you can always go on any of them because they're all clustered together. So I can go on RPY-01 and do a list and I'll see what's going on on the others. You can actually see some snapshots that started popping up now. The first one has two because I created one by hand but the C2 and C3 already got their first snapshot. And there's a good chance that C4 is running now. Nope, it's in our state. It's probably being unpacked and it's good. Okay, it's done. So if we do it again, yeah, it's running. All right, let's go back to RPY-03. So we saw that. Now let's try virtual machines. That's where things are gonna get a bit slower. Let's first disable Secure Boot. Usually not necessarily a great thing to do on Intel systems but because we're mixing ARM and Intel in this case and ARM Secure Boot is why we support it is not really supported by the vast majority of distros. The images will just fail to start unless you go and manually configure each instance to disable it. So we'll just turn it off globally for now. And we'll launch. So this is gonna launch an Ubuntu 20.04 image. Call it v1. It has the dash.hvm flag which is all that it really takes. Actually let's me use the same syntax as earlier which was 20.04 just to make the difference more even a bit clearer. So the dash.hvm says we want a virtual machine this time. Two CPUs, two gigs of RAM. The image is larger because as it turns out the kernel firmware and bootloader accounts for the bulk of the, usually accounts for almost as much space as the entirety of the root file system. So you often see virtual machine images about twice the size as container images and that's despite the package selection being identical. It's just the way it is unfortunately. It also takes a bit longer to unpack for a container. The image is effective is usually a squash of S which can be partly unpacked in parallel quite fast. And it's just a bunch of files that you write to disk and make it through the page cache on Linux for caching. For virtual machines we're dealing with a disk image. It's a block image of 10 gigs of five gigs, yeah, five gigs for images which is quite heavily compressed but still needs to be unpacked into a new block device which in this case gonna be about 10 gigs large. And once it's unpacked we need to do some partition table reshuffling to make it properly line up and make the GPT partition table make sense. So that always takes a little while. It's quite a bit faster on Intel. But on Raspberry Pi they're not known for extremely good single core performance which is what we're doing here. And in this case let's try to keep guessing. So it's not gonna go up by zero one because it has two instances. It's probably gonna go up by zero two I think. Let's go see. So if I go on up by zero two and then look at the process list. Yep, bingo it's this one. We can see QMU image DD that's running. So that's what's going to unpacking the compressed image into a ZFS volume. Similarly to with containers if you create again it's gonna just reuse that pre-created volume and it's gonna be very fast. It's the initial creation that can take quite a while. While that's going on let's start another one because that should land on another machine.  So we might as well create two at the same time and just save some, save ourselves a bit of time. So in this case it's gonna be an Alpine image from the Edge channel will call that V2 and I would expect it to land on zero three. Alpine is a bit smaller than Ubuntu so hopefully this one will be reasonably quick. Up by zero three is returning. Yep, so it's created V1 and it's starting it up now. So it's probably something that's worth addressing. Raspberry Pi 4s as you might be seeing right now are totally capable of running virtual machines. They do have CPUs, I'm 64 CPUs with visualization extensions. And that's definitely what we're seeing here. We can do console on V1 which will attach to the live text console of the booting instance that eventually will lead to a login prompt. There we go. Okay, so that's working. Now, if we do a list, we see V1 is running. We see its IP address, we see that its network device is called ENP5S0 and it already got a first snapshot. So it's really working the exact same way as container as you can see. Because we've got the agent running inside it you can do an exec and get the terminal. And this is running inside a virtual machine. Now, V2 should be starting soon, hopefully. It's still unpacking. And it's probably been like, as I said, V1 is running on RPI02, V2 is gonna be running on RPI03. So if we look at our own process list, there we go, we've got QNU burning through as much CPU as it possibly can to unpack this thing as quickly as possible which we'll then be able to play with. Oh, it looks like we're done. Yep, creating V2, starting it. Oh yeah. So that was quite a bit faster. Way faster. Yeah. It helps when the image is a bit small. Okay, so that's running. Let's just, for the sake of it, just do another run of V1, let's call it V1A, no limits. And we know that V1 is running on RPI02. So, similarly to the containers, it just create another one on the same machine and just see how quickly that actually becomes. So the create should be pretty quick, not as quick as containers, but close enough. And starting always takes a little while because we need to actually generate security profiles, load them up in app armor and a bunch of other things and then spawn QNU which also takes it, it's time sometimes. But yeah, there you go. That's significantly faster than creating it from scratch. And because LexD manages image caching in the background, we'll automatically update images for you. If you pretty much always launch the same operating systems, you'll only see that time the first time around, anything after that's gonna be quite fast. Let's see what we have running now. Yep. And if I do V2 and I get terminal in the V2, oh it looks like we don't have any working agent in our pine, that's quite possible. So we want a, this one should be mostly booted by now. Okay, so that's, we've got containers with your machines and snapshots going now. Let's make things a bit more fun and play with the next as well. So if we do cluster list right now, we can see we've got three nodes, they're all running AR64, so I'm 64 bit. But let's add some Intel to the mix. So I'm doing the exact same thing I did on the Raspberry Pi, which is enable clustering, select the IP address, training an existing cluster, copy paste the IP address of any of the Raspberry Pi's and the password, wipe the data, it's fine.  I could actually go with a much larger size on those because the NUCs are NVMe SSDs that can hold a lot more than the Raspberry Pi's and also are significantly faster. But okay, first one is done, let's do the second one. Label clustering, yes, yes, training cluster, yep. And they're not crazy expensive these Intel NUCs. I've never looked, I've never really looked into them, but Yeah, it depends on the configuration. They're not too bad, they're definitely not as low a price as like a Raspberry Pi for sure. But they're also not too expensive, yeah. And they've got the advantage of being compact, which is nice to have at home. Yeah, all right, so that's done. And now let's look at cluster list again. Yep, we're done. So we've got the Serial Raspberry Pi's and now the two NUCs. And we see the XZ664, so they're not training arm. The other thing you can see in there is that because of the way consensus works, we only try to run a database on three cluster members. And in this case, there's three initial ones. So the two joining NUCs don't have any database duty. That will change if we have a reboot one of the Raspberry Pi's because that role will then be handed over to one of the NUCs while it reboots and will then only be transferred back once one of the NUCs reboot. Anyway, so that's done. Now let's launch, because so those two NUCs are empty, they don't run any instances right now. Therefore, if I was to do another launch of C5 Ubuntu 20.04, chances are that's gonna land on a NUC. And we see the image is being retrieved again. So it's being downloaded from one of the Raspberry Pi's instead of going to the internet because if we have it in the cluster, there's really no point in going to the internet again. And then launch. You might have noticed it's a bit faster. Those machines are faster than Raspberry Pi's. And we can see it is here C5 running NUC01. It's also worth noting that we've got a syntax to choose different columns here. So we can do columns and then say we want the name column, we want the state column, IPv4, and is it AF architecture? It is. So then now, and I think we want, is it L for location of the last used? Capital L of things location. There we go. So now we can see what we have running. Let's do T. Okay. So we can see what are the containers with our virtual machines, the architectures and what they're running. So in this case, we can see that C5 is Intel 64-bit running on NUC01 as a container. I'd like to see a PowerPC Raspberry Pi, please. Yeah. There was work on our development boards for Power. I'm not sure where that went, but it was a thing. Let's do, yeah, let's do a virtual machine targeted directly, I'm not sure why actually it's put to targeted directly at NUC02 because we know, again, it's the one, it's not running anything. Therefore, it's obviously gonna land on it. So let's just do it on NUC02. So this time we're downloading the virtual machine image on that second NUC, which is gonna take a little while longer because, again, we need to unpack the virtual machine image. So I do expect it to do a significantly faster job than the Raspberry Pi's. And one thing that we can do in parallel to kind of show how that works, if I was to say I want to launch Open2 2004 ARM64 and call that, what are we at, C5 now? C5 already exists, apparently, C6. Because I'm specifying what the architecture of the image is, it will guarantee it's gonna land on the Raspberry Pi, was if I was to just say, give me Open2 2004, well, you don't electricity what architecture you want, so it doesn't care. It's just gonna pick whichever system can run and Open2 2004, whichever system can run and Open2 2004 and just run it there. So that's how you actually force a given architecture is by specifying it directly on the image. So in this case, ARM64 most have landed on Raspberry Pi and if we do MD64 C7, that's gonna have to land on NUC. And yeah, no to no surprise, C6 is running on RPY03, C7 is running on NUC01. And our VM is still unpacking. The other thing that might be interesting is during initial setup, I've opted to use the Open2 fan. So what that does is it effectively encodes the host IP in the overlay IP. So here we can see all of the containers have 240, 15 or 16, then some number and then some other number. So the 240 is the prefix, that's gonna be always the same. The 15 or 16 comes from the IP address of the system. So if I do that way, we can see this one is on 172, 17, 15, 120. So all of its instances are gonna be running on 240.15.120.something. And if you don't believe me, we can just grab that part of the IP, 17, 15, 15, 120. And that's gonna be all the RPY3s. And the same is true for all of the others, which means the kernel knows automatically where to route the traffic between them. And that means that if I exact into say virtual machine on RPY02, so that's V1, and let's say we ping C6, which is a container on, well, let's do a knock, C5, which is a container on a knock. It just works. Normally, Lexity also generates a DNS for you. So you can do C5.Lexity. So that kind of depends on the distro as far as how much that works. Okay, it does work. And we should have a new shiny Intel VM running too. Yep, V3 is running. And if we go in there, this is running 5.4 kernel, open to 64 bit. So that's our shiny virtual machine. Okay, now let's just play with something that probably not everyone has, but it's always kind of fun to play with. I do have a SEF cluster here. So that's distributed storage, which we use mostly for testing for Lexity. But you could possibly also deploy that on your Raspberry Pis. So you could have your three Raspberry Pis run SEF and then use next tunnel drive to have distributed storage amongst them and then configure Lexity to use it. So let's do that. Let's do the script. Why isn't it happy? I don't know, I'm not sure you remember what the script does. Okay, I see. Close enough to what we wanted to do. Oh, it's because of SNAP. Is it SEF that does the right thing? It is, there we go. Okay, it's in the right script instead. So what that script does is it just configures my SEF credentials and SEF cluster details on the systems. Not much point going into too many details about that because that's very much going to be depending on your particular setup. But if I was to say it's now installed and if we were to, for example, install the SEF tools on, let's do one of the next. SEF command just installs the tools. It doesn't actually install any of the daemons or anything because we're just a client to that remote storage in this case. And now I can do SEF status. We can see what we've got running. It's usually a bit of a mess because as I said, that's the cluster we use for testing. So our CI keeps creating stuff on it. But it's working and I configured to use it. So now that we've got that in place, let's tell LexD to use it. That's the part that's a bit complicated because storage needs to be configured on a per node basis and then once it's configured on all of them be enabled globally. So in this case, what we'll do is add this new storage pool called remote. Say it's blocked by SEF. Say what's the name of the SEF pool should be. And then we need to target that at every one of the cluster nodes. So we do R by one, R by two, R by three, back to one, back to two. Okay, so now they all know the source for it and then we can just do a global create which will have the first one actually reach out to SEF, create all the objects that are needed and then tell all the others that hey, you're up and running, you're good. So that's what we're seeing here. And because that cluster also supports SEF FS which is the, so what we did right now is SEF RBD which is product block devices. So that gets us, as the name implies, block devices. We can also create file systems on SEF with SEF FS. Let's go ahead and do that now. And again, do it on all the nodes, look at one, look at two. It's kind of annoying having to do it that way for SEF because it's gonna be usually the same config on all of them, unless you've got a really weird environment with different tierings or something on per-byte machine basis, but for things like non distributed storage, if you were to add like another pool based on ZFS or LVM or something, it makes a lot of sense being able to configure it on a per node basis to specify what the block devices for example, for each of them. In any case, this is created. So now we can see we've got three storage pools and next day we've got local, that's our ZFS but one that we've got everywhere that we configured during init. Then we've got remote which is a SEF block device and we've got remote FS which is SEF FS. What we're gonna be doing now is create a new volume. So we'll create a volume called shared on the remote FS pool and we'll be using shift FS to make it attachable to as many instances we want at the same time to security shifted true. That volume is created. We could actually go on volume set. Let's actually configure like a size on it just for a bit measure. Let's put it at 15 gigs. There we go. And now let's modify our default profile to attach that particular volume. So as we'll do, we'll add a new device to the default profile called the device shared. It's a disk. It comes from the remote FS pool. It's the volume itself is called shared and we want to mount it in slash shared. Okay, that's done. So now let's look at how things behave. Again, pull our container list and let's go and see one and let's look at what the slash shared looks like. Hey, 20 gigs, it's mounted there and this one is on RPI-01. So let's do, well, we create a five code foo and we just echo the name of the container, I guess. Let's do that. Okay, let's do C2. Where is C2? C2 is also RPI-01. So we'd expect that to work for sure. Yeah, that's right in right C2 in it for just for gigs. But now the more interesting thing is that let's use C3. C3 is on RPI-02. It's not on RPI-01. It's also a different distro. I can't remember what that was. It might be Arch Linux or something. Yeah, that works fine. Okay, I mean, I could go on and on and on and show that it works everywhere but let's maybe just pick a NUC and C5 is a NUC. So let's go at C5. Again, works fine. Now for the tricky part, virtual machines. So if we go in V1, you might notice the lack of a slash shared. That's normal. As I mentioned earlier, virtual machines not very good at being updated live. Not sure that V2 will actually restart because it seems a bit stuck. Let's just stop V2. Let's just restart the rest. So we restart V1, V1, and V3. So as I mentioned earlier, virtual machines we can't easily do hot plug for everything. We're working on that. There's no good reason why this one we couldn't do hot plug other than we didn't get around to it. We do have the agent inside the VM. So LexD could have told the agent like, hey, there's a disk, please do something about it. But we don't yet. So restarting the VMs. We can do console on V1. Okay, let's start it. I'll just get that shell inside it. So just need to check something. All right. So shell inside the VM and slash shared. Will you look at that? Neat. I can write V1 in there. We can see it also shows as, I think she shows 26 gig, which is slightly wrong. I'm not sure why that one doesn't show the right size. There's a lot of different layers to actually expose it to the VM. So it's not too surprising that it might be a bit confused. Oh, no, nevermind. Oh, I set it to 25 gigs, are you all right? Anyway, don't remember what I did exactly, but okay, so it does see the same thing as the containers, same size. And we do see that shows us as we want. Oh, actually, I do remember what's going on. The Ceph version we use for the test cluster is still on Ceph 12, which does not implement the full quotas that we get in think 14 or 15. We've got another test cluster here that does it properly, which is why I was a bit surprised anyway. So you can actually, with Ceph FS, you can actually start and share disks, not only across containers and virtual machines, but also across the entire cluster, which is really quite neat. The other thing, so if you were to just do it locally, you can still create a volume and you can still share it with whatever is running on the same machine. But obviously if you try to attach it to something on a different machine, it will work so well. And lastly, let's show starting a container on the Ceph volume. C6, what's wrong with this one? Did we use C6 already? Oh, yeah, we did. Maybe I'll see, yeah. C8? No, that would work. Okay, well, let's go and target something specific, maybe. All right, images. Do I make a typo somewhere? Is it the name of the Ceph cluster already, Seth? It is not, that is a very good point. There we go, that should look better. And so in this case, now what's going on is that we are unpacking a new volume on Ceph for that particular image and then we'll be creating a, in this case, a new container from it. That was pretty quick. Where is it running? Is it running on ARM? Or is it running on Intel? C8 is running on Intel, it's on a Mac. Okay. And we can do the exact same thing with virtual machine. So let's do V4 and yeah, that should work fine. No idea why that's gonna land. This one might be quite a bit slow. While that's going on, let's say, I'm starting to wonder what V4 was the right name. Yeah, V4 should be fine. Okay. So C8 is the container we just created which is running on Seth. We can see its root device here is attached to the remote storage. The cool thing with that is it's very cheap now to move it. So if we were to stop it, now we can move it to a new target. Because it's Intel, we obviously can't move it to a Raspberry Pi that wouldn't work so well. But we can move it to the other Mac and it's done. No data was actually moved in the process. The only thing that was changed is just a database entry and a couple of empty directories on the file system, pretty much. Which is significantly faster than the alternative which would be transferring the entire thing. So if we do C5, for example, C5 is stored on ZFS. Let's move it. So C5 was on the quantum. So it's going to go the opposite direction. As I said, a bit slower. Just a tiny bit. Yeah. That should still succeed unless we've got some weird kind of case I didn't think of. But yeah, it can be quite a bit slower and uses our normal migration API. Hey, did it complete? It did. Right, so that did take a little while longer and the initial startup would be a bit longer too because now we need to actually start that dataset and start everything back up. It does work, obviously. You can move things around just fine but it is significantly faster when your storage is distributed. The other thing that's interesting that I can't super easily show here is where you to lose a system that's got containers or virtual machines stored on ZFS. So if it just goes down, loses power or something. So long as your cluster database functions, which it should because it's replicated, you can go and move those instances to another system even though the source system is no longer available. Again, because there's really nothing there. Whereas in the other case, you obviously can't do that because you can't access the source storage so you can't move the actual bits. Okay, I'm seeing a notification in the first time and also there's a good chance that our V4 is starting. It is. Yep, so V4 is a virtual machine on Raspberry Pi 01 which is gonna be backed by Ceph. Oops, that'd be too, V4. Yep, remote disk. And the other thing that can be quite nice we'll just wait for it to do its thing. Let's just do a console on it. Just see a boot. The other thing that can be quite nice here is that Ceph can give you, because that just as the external one is quite large, give you a amount of storage that you couldn't normally do. So let's do, let's stop it and let's do config device override V4 root size and let's do 80 gigs. Oops, not override, it's gonna be a set in this case. Okay, and as a reminder, this is running on, where's it running? V4 is running on RPY1, that's the other same. So the Raspberry Pi only has 29 gigs of storage. So normally running a virtual machine with 80 gigs, not really an option, but yeah, because this is using remote storage, there's nothing wrong with it. Can't wait for a console. The last thing I just wanna show before we wrap this up is, so far I've been doing all of that stuff from the systems themselves, but now let's just move back to my laptop and let's add a new remote in next day. So I call it demo and let's do the IP of that cluster. And now you can do, so if I do an Excel list, I'll see my own laptop, but I cannot switch remote in next day. So we can switch over to the demo remote and if I do list here, now I'm seeing the remote cluster from my laptop. We're gonna see that that V4 is currently starting. Yep, and it's starting with an IP and everything. So if I go console, oops, doesn't read it in, oh yeah, we would have a console open on it. But I can get a terminal inside it and we can do partitions. And that shows that our list is 90 gigs large. So that worked fine. And for one last trick to show, Lexity supports attaching to the VGA console of virtual machines as well now. So I can only do that for my laptop because obviously there's no graphical redirection on the thing going on in the Raspberry Pi. So I'll do it on one of the instances now. What can I do that's, let's do Intel because the graphic I put them to work better. NUC02, let's do V3. So I'm gonna be restarting that instance called V3, oops, let's not type it's console. Okay, so this tells Lexity to please restart that console that instance called V3. And once it starts back up, automatically attach to the VGA console and show that to us. And I can show you, it's just did it. I just need to move that over to there. This is unfortunate. No, there we go. So we've got the Lexity branded firmware. And this is effectively Ubuntu 20.04 starting up on this one. And it's probably done. So if we go and send key and switch to one of the dominoes. Yeah, there we go. I've got the login prompt. And again, that's working. That cluster is running in my basement, I'm on my laptop and I can use the Lexity cluster and its API to do things like attaching to any of the instances, remote console, all of that stuff and even VGA console in this case. All right, so I think that's it for the demo. Let's move back to some slides and wrap things up. Nice. So thanks to Pan for the demo. That was really cool. Like seeing stuff like that, it was really exciting. I might just build my own cluster right here. So let's quickly recap. I mean, we're almost out of time. Choosing system containers or virtual machines. That's a tough question. Advice that we can give is pick what's best suited for your workload, because the management is really identical and we also have a wide variety of images available for both types. Obviously with system containers, you will have less overhead than you have with virtual machines. That's something to keep in mind as well. But it really depends on your workload and probably also how much isolation from the host you really want. In terms of security, it's a good question whether containers or virtual machines are more secure. There was always dictum that virtual machines are more secure than containers. I think with the recent attacks and insecurities that we have seen both in CPUs and in general in the kernel, this is not necessarily true. But yeah, so base it on your workload. If you're really interested in scaling to a large number of machines, and we really made sure that this works out of the box, we have built in clustering. So lexity essentially, multiple lexities act as a single lexity or as a single entity that is easily to manage. It's highly available and it's really easily scalable. And especially as Stefan has demoed, if you combine it with SEP, you get fall tolerance and also you get a few nice other properties, such as for example, when you move virtual machines which tend to be way larger than system containers most of the time. Between different machines that are on the same SEP cluster you get basically an almost instant move. We have easy storage network, GPU and generic device pass-through. The led tabs too, mostly relevant for containers at this point, but we're working at making this even easier and advancing this also with virtual machines. And there's a lot of work going on this front. So you can at least for expose hardware easily to containers at this moment, you can use quotas and limits to prevent abuse. This obviously applies to virtual machines and containers alike as you have just seen in the demo. And we work everywhere. So not just do we run on a wide variety of Linux distributions, we also run on all mainstream architectures, all the ones that are still supported. I mean, we're basically just limited by the Go compiler if you look at it from the lexity perspective. So if you wanted to run on alpha, that probably wouldn't work but simply because Go doesn't support it, there's technically nothing that would prevent us from actually doing this. But if you're talking at 390X PowerPC ARM x86 that all just works out of the box. And we also, and this is pretty nice, we have a client, our client tool that talks to the lexity daemon that is available for Windows and Mac. So you can have the lexity daemon running on a dedicated server in your basement but you can still use your Mac or Windows to talk to it. All right. And because of this demo, you don't have time to do this but I could have showed being on a Windows system and that last step I did while I added the remote cluster on my laptop and then attached to the graphical console. I could have done that from a Mac or from Windows just the same. Indeed. And important to know it's production ready and we really mean this. Not just do we do long-term support for five years and we've done this for a long, long time. So we've already done this with lexity and we still have versions of lexity that we actively support and backport fixes to. This is really important to us and so something where we have been reliable for a long, long amount of time. I mean, almost like over 12 years by now. Lexity is younger. It has been around for over four years but it follows the same model. We maintain long-term stable branches that we support for five years that we take care of that you can rely on that will only see fixes, no features that we don't regress anything. We're very responsive to issues. So if you come to us and point us to something that you need fixed or that you think is a bug, we will be quick to respond and fix it. And as you see, we also run production, Chrome OS, Travis, there are a wide variety of other customers and users and users of lexities and we're pretty proud of this. And so yeah, I hope this whole workshop gave you an impression of how you can run your own home cluster and you have learned about how we use containers and how we think system containers and virtual machines basically tell the same stories with a different twist. Stefan, do you wanna say a few closing words? No, that's pretty much it. I was gonna say for the LTS, we currently support 2.0, 3.0 and 4.0 of Lexity as LTS branches in parallel. We only do active bug fixing on the latest one. So 4.0 is getting pretty much weekly or sometimes even daily back ports of bug fixes to it and then we really start every few months. For 3.0 and 2.0, they're in security on the mode at this point where we keep them buildable, we've done some minimal updates to keep them working on your kernels, for example, but otherwise they're only supported from us as far as fixing security issues if any should show up. That's pretty much it. If you're interested in paid professional support or whatnot on next day, works best on Ubuntu, you can, because we work for Canonical, you can get Ubuntu Advantage that covers Lexity and that also gets you a few extra features like integration features that Lexity supports, for example, integrating with the role-based access control service that can come on some Ubuntu departments. If you've got that set up already, then Lexity can integrate with it. But otherwise, Lexity works the same everywhere on every distro or even Chromebooks, Windows, Mac OS. And that's it. I think we're just about out of time. So if you've got any questions, please go ahead and ask around to answer them. There's our website there where you can try Lexity for yourself online. What is also linked to our forum where we've got tutorials and a bunch of very helpful people that can help you set up your own lab, goes through with you on the best storage or network or whatever setup for your particular application. And yeah, thank you very much for attending. Thank you. Hopefully we'll see you around. Yep.