Welcome back to this session. This time I'll be talking about transactional updates with B3FS. My name is Ignat. I'm working for uh, SUSE as a research engineer and I'm the maintainer of a tool called transactional update which is implementing basically what I'll be talking about now. So, the question is, um, why do we need a tool or a mechanism like that at all? Because, I mean, most distributions will not claim that the update mechanism is not reliable. I mean, nobody would claim that. Um, but in reality, we see that things can go wrong. Still. Just a second. Yeah. Why? Because, um, packages may be failing to fail during installation. Uh, just take the example, uh, you are installing an RPM package which has a post script. The post script will fail and then you are in a situation where you don't know what state your system is currently in because the system has modified, uh, the package has modified something, but you don't know how far it got and what it actually modified. Uh, that leads to several people having a fear of distribution updates. We often hear that we don't want to update, keep the system running as it is because things may break and yeah, we don't want to have that. Um, that's especially true if you're, for example, if you are thinking about a desktop system, uh, about diverse updates or if you are updating your complete desktop environment, how would do you do that live in a currently running session? Because you're updating yourself, the systems you're currently using, uh, live while you are using them. So that's also not an ideal, uh, uh, ideal way of doing updates. Um, if your package is providing a service, you'll often get, um, restarts of that specific service, which is also maybe not what you want if you're doing that in a production system because the package is controlling when you, when it's going to reboot your things. Um, and of course, if you're doing that as a maintenance, in a maintenance window, um, you have quite long down times until that update has finally finished performing all of its operations, which may also not be ideal. If you're going one step further and thinking about atomic systems or transactional systems, uh, we had them a few times, uh, during this conference already, uh, then things are even worse because you want to automate a lot of things as efficiently as possible. You don't want to have a lot of down times and automation is really becoming to start getting difficult if you can't know what state your system is currently in. So you don't know how can you get out of a mess if the mess started. Um, also, one point I also wrote down, uh, for embedded systems, uh, you can't really afford it to break the system of the user. If you break it, then you really have a problem. So, um, that's why transactional updates are there and transactional updates are not a concept we invented. Transactional updates can, uh, over various distributions, uh, be defined as the following in, uh, transactional update is an update that is atomic. So it means that it's either fully applied or it's not applied at all. Um, and it must not influence the currently running system. And the second point is, uh, it has to, there has to be some option to roll it back. If you realize it didn't do what it, what it was expected to do, you have to be able to roll it back and restore your previous state. Um, this slide contains a lot of logos because basically that's all, all systems which are mainly implementing, uh, things like that. Um, and most systems have some things in common. First of all, uh, they are having those transactional updates I'll be talking about, but you can see, uh, some more similarities. Uh, most of them share root file system, read only root file system. So you can't actually write back to the root file system. Um, most of them are either intended to be used in cloud or image environments or in embedded systems. Uh, they are usually consisting of a minimal base system. Um, some of them have automatic reboots and updates. And some of them also provide integrity, uh, protection. So, um, those are the basic concepts that common, but the implementation of the distributions is completely different. What I'll be showing now is the way B is open SUSE in that case have implemented it because, uh, that's one way of implementing it. And it's using BDRIF S as one of the main components. I guess most of you know SUSE is one of those, uh, distributions which is using BDRIF S by default. So, uh, one question for the audience, uh, who is aware of how open SUSE is working? Who is using, uh, SUSE or open SUSE systems? Okay. Only one of you. So, um, yeah, let me get a brief introduction on, on to, uh, how systems are working. Uh, for package management, we have SIPA. That's basically apt-get or, um, uh, DNF in Fedora, um, which is all doing all the packaging work. Uh, we have SNAPER. SNAPER is a snapshot creation tool. Um, we are using BDRIF S. So, uh, SNAPER is one of BDRIF S, one of the really cool features of BDRIF S. And SNAPER is automating that. So, whenever you are updating a package, you will also get a snapshot before and after that update. So, you can just roll back if anything goes wrong. I'll go into detail in a few slides. And, of course, one of the most important ingredients into that variation of transactional updates is using a, let's say a file system which is able to do snapshots. In our case, it's BDRIF S. In theory, uh, the same mechanism could be adapted for other file systems, but, uh, there are a few caveats if you really wanted to do that. So, how does that actually look like if you're using an open, uh, uh, SUSE distribution and just take all the default values during installation? You will get a system installed and whenever you are updating, as I said, SNAPER will create a pre-snapshot, then do the update within the currently running system. That's snapshot number two. Uh, that's point number two.  Um, one of the problems, as I said, is all of those operations are done in the currently running system. So, if anything goes wrong, you're stuck, you have to roll back to one of those snapshots. And, yeah, uh, that's usually not what you want to do, uh, deal with, uh, how to get, uh, back from those snapshots. And if you have a look at the definition of what we wanted to achieve, uh, with transactional updates, we can see we have that option to be, uh, to roll back if anything goes wrong, but that atomicity we wanted to achieve is not given with the mechanism we currently have. That slide is not supposed to be in here. Um, third one is, um, with transactional updates, we will change that behavior. With transactional updates, we will still have our currently running system, which is in this blue snapshot you can see here. And we will then just create another snapshot next to it based on this system. Um, the update itself will be performed in that new snapshot. Uh, the currently running system won't even notice an update is performed somewhere else. And, of course, that has a huge advantage. All your system services are just continue, uh, will just continue to run as they are. And, yeah, the update itself will be running on the background. If anything goes wrong, then that new snapshot will just be deleted on the background again. The system will just stay where it is and everything's fine. That's the basic concept. Um, what's happening with the, uh, update actually is successful. Then the new default B3FS snapshot will be set, set to that new snapshot. So whenever your system is rebooting, it will reboot into that new snapshot and use the updated system. Uh, one of the basic elements of atomic systems is that reboot part because you can't guarantee atomicity in a live system or in a running system. That's also one of the things which is common with, um, the other atomic systems. Uh, yeah, that reboot part. Um, we also have a helper tool called Health Checker. Um, if the system was actually rebooted into that new system and anything goes wrong in there, it will try to detect that. For example, you can check for services which, which services which are not running as you want them to be. You can check for files which may not be there. You can just put in a script which will check everything and check if everything is running as it's supposed to be. If it's not, it will just set back the default snapshot to the previous one and reboot the system again and you are back in your previous working snapshot without any manual intervention. Uh, that's the whole idea of it. Uh, we also have a Crab 2 snippet, um, which would be better if it was solved by, um, uh, there's an, we have to do it manually. SD, uh, boot by system D would be able to do that automatically, but we are not using that. Um, if the system does not even pass in an ID, we'll also detect that and also do the rollback from Crab 2 itself. So if you want to just have a quick look at how that actually looks like, um, I'll be using micro s. It's a part of the cubic project, which is doing that on the open suicide. Um, you'll see that it is a read only root files. It has a read only root file system, but still even with a read only root file system, you have to be able to write to a few directories, mainly VAR because obviously it contains, contains variable data. You have to be able to write that, um, and it is C because probably you don't want to use the default configuration all the time and want to change a few bits occasionally. Um, I don't have too much time, so I decided to record a few things and play them as a video. What you can see here, uh, yeah, it's working perfect. What you can see here is a default micro s system which just booted up. I just increased the font size. I hope you can read it, even from the back now. Is it visible somehow? No nodding, a few nodding, perfect. Um, we can see our super command. Like I said, it's the default packaging command, uh, and I've just listed the default repositories. You can see those are the default tumbleweed repositories, tumbleweed is the rolling release distribution of OpenSUSE and we can just use super search to actually search for some packages. You can't use super anymore as soon as the actual file system would be modified. In this example, I'm trying to installing, to install something and that obviously breaks. We'll get a warning that this is a transactional system and we have to use transactional update to modify such a system. We can also confirm that by just touching a random file, which simply won't work with a message read-only file system. So let's disassemble that system a bit so we can see how such a system with a B3FS file system layout is actually working. I just typed in find mount here. Uh, find mount with B3FS and overlay. Uh, and we can see a lot of sub volumes. First of all, can I highlight this somehow? No, I can't. First of all, on the top here, we see that the root file system is read-only as intended. We have several more sub volumes here, or better here, um, which are all writable, read-write, which will contain the variable data, which we need. And we could have seen about the, that there was a snapshot directory. I can, we can see that in the B3FS output here, um, which will contain the actual snapshots. We can see someone decided to modify the system quite a few times. We have 10 snapshots already and the snapshots directory contains the previous system snapshots. When typing in Snapper list, I can get a list of all the, uh, changes and we can see a bit of what actually happened. I mean, we could also see that from, uh, uh, interpreting the, the B3FS data, but Snapper will just give a better overview or more readable overview of all the data. And we can see, over here again, sorry, that we got some snapshots. Um, it seems something was wrong with snapshot number seven because it was never used as a base, so someone seems to have rolled it back. Uh, we have also, uh, two times snapshot number eight. And we are currently in snapshot number 10 marked by that star. Yeah. Talking about the default snapshot. Um, if you're having a look at, uh, a conventional root file system, you usually have slash as the root. This is not the case in our system or in our setup, uh, because the root file system is basically one of those snapshots just to make that clear. Now let's install a package. I'll try to install TCP again and we can see some output by transactional update itself and then adjust the regular super output. So we don't have any special dedicated mechanism, uh, like most other distributions to which are having their own packaging format, but it's just using the default RPM files. I'm sorry. I guess I shouldn't step on that. Um, we just, uh, using default RPM packages, which are part of the tumble, open SUSE distribution. Now it's just going to install that. I'm sorry. The network was quite slow on that day and it's finished. Now, uh, we'll try to execute TCP ramp and we will notice we can't, it's not there. Why? Of course, because it was done in that snapshot, which is still not visible. And if we call snap list again, we can see that a new snapshot was created, which will be the default on next boot and we'll just simulate a reward now, just fast forwarding it. And we can see, uh, TCP dump is working now. It's for the virtual machine, so we don't get any traffic here. Transactional update has lots of options. Um, for example, uh, basically it's just replacing all the writable parts of the super command. So we can see here we have dub, we have up, we have all those package commands. Um, and we can roll back as we want and a few special things which also require modification of the root file system. Now the interesting thing is, um, can we somehow debug something or what if I have more complex things to do than just install a basic package? Then we have that transactional update shell command and I've combined it with an installation command again. In this case, we'll just get IO top installed again, not again, we'll get IO top installed and the execution version then stop in a transactional update shell. We will see that in a minute here. We'll know in a transactional update shell and we can use IO top immediately. Why? Because it just CH routed into that new snapshot. So you as an administrator, for example, are able to check that the system is, did everything or the updated everything as it was supposed to do. Now imagine if you have more complex update procedures. You would just open a transactional update shell and then do the update in there. You can check all the, all things you have to do. Um, and only when you're finished, you would exit the shell and only then you would reboot and get all the updates done once by simply rebooting without any downtime in between. It's starting to flash. We can still see something. Now the interesting thing is, what if I install a package that doesn't work? It's not always that obvious that it's called break me. Usually things start to break more subtile like that flashing thing here. Um, in that case, the post script or the prescript even of that script failed. We will just get the message that super was not able to install this successfully because exit code nine and in that case, it means it was a manual abortion. Uh, but, uh, you'll get one code 108 or something like that. If a script has failed in any case, the snapshot will just be removed again and the system is just as before. We won't get a new snapshot. If you have a look at our list of snapshots, we see nothing appeared here. So, um, that's just a cheat sheet in case you want to know all the commands, uh, at once. And that's basically how such a transactional system works. Seems easy, right? Um, I know some of you are distribution developers also developing atomic distributions. So, yeah, it's not as easy as it looks like. So let's have a deeper look at what's actually happening in the background. First of all, we have that ominous war handling. Um, with the war directory, we have one problem. Namely, it can't be part of the snapshot because if you would snapshot that directory, yeah, it would be static and let's assume you have your database in there, then all the data between that snapshot and the next reboot would just be gone. That's not possible. Uh, you usually also store your Docker images in a separate, uh, wasp volume, uh, which is also not something you want to snapshot. So, why especially in a case that it can't be rolled back. Um, and it also mustn't be mounted into the, uh, CH route environment where, where you are updating the system. Why? Because then you would modify the current state of the system, which you also don't want. So, how to deal with that? Uh, the solution is pretty simple. You have to create system D services. Usually if you're using system D, otherwise, um, SSV services, um, which will update or modify things on the next boot if it has to be modified. Imagine a database update, which gets a new database scheme or whatever. Um, then the database itself will be updated during the transactional update run, but the migration of the data will only run, um, during the next system boot. So, yes, there's a slight inconsistency here because not all of the data is roll backable, roll backable, roll. Can we roll it back? Um, can we roll it back because, yeah, uh, you have to have that production data in a separate partition and, uh, care for it, uh, yourself, basically. Uh, we all have the other case of ETC. Um, usually you as an administrator or your configuration management software will have to be able to modify files in ETC. For that reason, uh, ETC is mounted as an overlay file system in OpenSUSE. Um, the data itself is again stored in, in VAR. The data of the overlay file system is stored in VAR, so that's another reason why it can't be, why we can't make snapshots of that. Um, and, what did I want to say? Yeah. And, uh, that snapshot, uh, those snapshots will be stacked. Um, that means, um, if you are updating your system all the time, you'll get several overlay directories in VAR. First we'll start in Valip O, Ovalay snapshot number, something. So if, let's assume your configuration management software changed something, but the update also changed something. Then the newest overlay will actually win. In that case, the, uh, the, um, uh, the overlay of the package update because that will get the new snapshot number. Um, that's, uh, in case of you, in case you visited also the ETC talk this morning, that's one of the reasons why we want to have a separation between, um, package data in ETC and data which the administrator actually changed. So, um, yeah, you wouldn't get conflicts in that case or not so often at least. Uh, but usually it's just working fine. Uh, the newest file, uh, the filing of the newest overlay will win. And basically we are in a consistent state. Uh, in my opinion, that's better than, um, for example, three way mergers, uh, which, uh, for example, uh, core SS doing, uh, because we always have a consistent state in itself. Um, but yeah, that's, that can be discussed probably. Uh, we also have a few other directories, uh, namely opt, which will contain optional data. That's not part of our root file system. If you're installing something in there, it will just be installed immediately. Uh, we have VAR lock. Why? Because you probably still want to lock what's happening during the update. And we will have, uh, boot grab to, uh, mainly on, uh, UFI systems or on bio systems, you have to get, uh, the boot sector somewhere. Uh, you can't roll that back if you updated it once. Um, just to be aware, everything will, uh, apart from those few directories, nothing will be mounted into that, um, update environment, as I'd call it, uh, including SRV, because SRV shouldn't contain any user data or any data done by packages anyway. Um, so yeah, you basically have your core system, which you're updating. And then you can build up on top of that system, for example, with Docker containers or whatever. Um, but you have that core system that will always reliably be able to update itself. Another tool we introduced, uh, is a tool called Reboot Manager by Reboot Manager. Uh, we want to have as much automation as possible. Um, so if a reboot was successful, um, Reboot Manager, uh, sorry, if an update was successful, Reboot Manager will be triggered to reboot the system. Now imagine you're having a whole cluster of, uh, different nodes. And, uh, yeah, usually you don't want to have all the nodes going down at the same time, because accidentally the update was triggered at the same time. So Reboot Manager will have several strategies, for example, by synchronizing via at CD, uh, to synchronize the reboot, um, of the system. Yeah, uh, that's it basically. Um, there are a few pitfalls, which are current limitations, which we are trying to solve, uh, currently. Uh, the next snapshot or the snapshot will always be branched off the currently running system. You can't go back and say, oh, um, I've just rolled back and want, uh, still want to use that new snapshot as a base. That's currently simply not possible. Um, and you should be aware if you're using, ever use, if you should ever be using an open source system, which I can really recommend, uh, then, uh, transactional update is the only mechanism to, uh, update a read-only root file system. You may not need that, but if you do, transactional update is the only way to do that. So, why are there slides? That shouldn't be in here. Okay. Um, so, uh, availability, as I said, you can, uh, use, uh, open source, uh, tumbleweed as a base, uh, or open source, uh, Leap 15 or 15.1. Uh, those will have a transactional server role. Basically, what you get is a conventional trans, uh, uh, open source system, but with a read-only root file system. Uh, we have open source micro s and open source cubic. Those are the, uh, operating systems intended for container use or in the case of micro s for one single purpose. So, you can use the, the base and build upon it, whatever you want. And on the commercial side, we have the Susie Cas platform, which used to use transactional update. It currently doesn't, uh, they are, they free-based it on our new stack. Uh, but it will be, uh, supporting transactional updates again. And we have the Susie Linux Enterprise Server 15.1, which is basically the same as with, uh, open Susie Leap. You will just get a transactional server role there. Uh, if you're interested, all the, um, documentation, all the links to the, uh, sources and so on are on, uh, cubic.opensusie.org. And, yeah, if you're interested in the, the technology in general, uh, and may want to adopt it to your distribution, you'll also find, uh, the necessary links in there, or you can just talk to me or, or my, uh, or a seed channel or whatever. Uh, and, yeah, that's should be it. Do you have any, yeah, lots of questions? Thank you. Uh, several questions, uh, short questions. Uh, first of all, uh, does, uh, does this approach need support from InetramFS? Um, not really. We do have, uh, InetramFS module to, uh, get a list of files which will be conflicting because, as I said, um, not everything is mounted in, uh, in, uh, sorry, in the root file system. And, uh, Etsy may have conflicting files. So we are just comparing those files which timestamps would match, uh, would, um, indicate that there are conflicts and they are just printing out warnings to the syslog. But apart from that, no, there's no requirement for an Inetati component. Okay. Uh, sorry. Uh, next, uh, next thing, uh, you're talking about a grab to a configuration. Do you have integration with, uh, different bootloaders over the, with loaders? No. And, uh, that's, yeah, what, what's the bootloader would have to do is, uh, it would have to be able to boot from the correct B-DRIV-S partition, namely the default partition. If it's able to do that, which will probably be the default, be the default anyway, then you're basically fine. Then everything's just working out of the box. But you use subvolume for root of s, right? We are using sub volumes. Yes. But we are setting the sub, correct sub volume as the default B-DRIV-S sub volume. So, um, yeah, basically if you're just mounting B-DRIV-S with mount, for example, you will get that sub volume anyway. And if the bootloader is doing it in a similar way, like rub is doing, uh, then it will work out of the box. I have no idea about sdboot to be honest. Uh, I have not even a, uh, idea how it's B-DRIV-S integration works. So if it does need something like that. Okay. And the last question, uh, what is the criteria of successful update? A criteria for successful update is that super didn't return an error in our case now. So it's before the reboot. I mean, if you reboot with updated, uh, system and, uh, something goes wrong. Uh, yeah. Then we have health checker, which is trying to detect those, uh, cases which when the system did not come up correctly. For example, your database is not online or you may check for, uh, the, uh, conversion scripts now that I'm talking about databases. Uh, if you have, uh, a database migration, which would have to be done. You can also make a check and health checker, uh, and see if the database was migrated correctly. If it wasn't, health checker will automatically roll back to the previous date. If health checker doesn't boot or if B-DRIV-S itself has an error, then we have a problem. That's, those are the two components that mustn't break. And, uh, in a partial, uh, partial way, uh, the part of Crop, which is installed to the MBR, of course, or to the, uh, boot sector also mustn't break because, yeah, then we have nothing which can boot anymore. Uh, as, as a developer, is there a way that I can test my new packages without rebooting the machine? Like, can I, hey, I want to test my database in the new container beforehand? Because like, spending a couple minutes rebooting is really tough. Um, if you're using containers anyway, uh, you don't have to rebuild it at all. You just maintain your containers. Er, sorry, when, when I say container, I meant the, the like, the new system image snapshot. Um, you can use the transactional update shell, um, because you are basically inside your new system, but you won't have your Vasa volume mounted there. So, if you want to test something which depends on var, then no, you can't. If you don't, then you can use that environment for testing everything else you want. Yeah, hi. Uh, we have an embedded system and we have, yeah, something like this, but not on the file system or level, but there we have a network, uh, network API that you have to, to call from the outside. And if that's not called in a specific specified time, then you reboot the system. Do you have anything like that here? Um, do you mean, uh, you did, you did the update already and then boot into the new system and realize all the network isn't coming up anymore? Yeah, for some reason, the outside can't reach you anymore. Um, the question is, can you do that with health checker? I would say yes, because, uh, if you use, for example, uh, the network manager online, NM online, no. What's the command? You can use, uh, there's a command for checking if the network is online already, or you could just have a loop and checking for the, uh, for the network. If it doesn't come up, then you would have to return with a non-zero exit code and health checker would roll back the system. So, basically, you could just put any, anything which is scriptable, can be put in there. Um, or I mean, you could also write your own service or whatever, uh, you are using and yeah, just do the rollback manually or call, uh, rollback helper manually. Uh, sorry, transactional update rollback manually. I have another question about the health checker. What dependencies does it have? Is it agnostic enough to be able to use it on, on another system, on another atomic system? Yeah, but don't look at the source code, please. Uh, there's some really crude, crude text to get, uh, grub to, to be able to do that rollback mechanism. Um, that's why I was having a look at, um, SD boot from SystemD because that has support for, uh, that already, it doesn't need those hacks. Um, health checker itself is just a shell script. Okay. Or a collection of shell scripts for grub, for, uh, SystemD, and for, uh, yeah, basically that's it for the shell scripts in the, in it already. So a clarification, uh, when you're rebooting a URL, you have some migration to perform on var. Do you also create a snapshot there? No, um, everything which is done on var has to be taken care of the system administrator of the, uh, why not? Because let's say I have two migrations to perform on, let's say, two databases, the database system, one is successful, one fails. What happens then? We're not just having a separate snapshot for var so that you are able to rollback if something goes wrong at that phase. That's a good idea from, from that perspective. Yes, I agree, uh, but we are not doing that automatically because I don't think we can detect, uh, whether we need to do a rollback, uh, sorry, we need to do a snapshot because we don't know what will happen during a boot. We just know it is a new boot after system update, but we don't know what will be going to happen.  Yeah, I mean, that's a good idea. We could just create a snapshot on boot and if health checker was successful, we could just delete that snapshot again or that's a good idea, actually. Anybody else interested in that? No, I have a different question. Um, so I have a different question, which is, um, the mechanism that you show is doing client side compose, like every single machine is downloading packages and applying them and snapshotting and all the stuff. Um, did you, there is another kind of like approach or way, which is, um, you do this kind of compose somewhere else, like remotely and then all the notes are just downloading, like premade, snapshot, let's say, did you experiment or try that? Uh, that's basically what OSTry on Fifidora site is doing, uh, but we don't like it at all to be honest because it's a completely new packaging system. You have to, uh, yeah, package all your stuff for OSTry, uh, with our system, we can just use the RPMs, which are there anyway. So, uh, yeah, uh, I think that's more, they're more flexible. Yeah, uh, we don't have time anymore, so thank you, uh, Ignace. Thanks a lot.


Welcome to Spotlight Advanced. I'm Bruce Gulland. And I'm Megan Nolette. Spotlight uses a special English method of broadcasting. It is easier for people to understand, no matter where in the world they live. A group of people sit in a theatre. A curtain hangs across the stage where actors will perform. The people speak quietly. Suddenly there is a loud noise. The people watching go quiet. There is another noise and another. They come faster and faster. Then just as quickly they stop. The curtain opens. A man stands alone on the stage. He balances on one leg. His arms are above his head. He wears very bright clothing. His face is painted in strange black and white lines. Music plays. The man begins to speak. This is the beginning of a kabuki play. Kabuki is a special kind of theatre from Japan. It uses a very different style of acting than most people know. Today's Spotlight is on Kabuki Theatre. Kabuki Theatre is a historic art form from Japan. There are special theatres where people only perform kabuki. Some actors are only kabuki actors. They play the same characters that their family has played for generations. But kabuki was not so ceremonial in the beginning. Instead, kabuki began as a kind of street theatre. The first actors did not even use a building. Instead, the plays took place on the banks of a dried river. Izumo no Okuni was a dancer who lived in the 15th century. She brought together these first plays. At first, kabuki was mostly song and dance. And Okuni performed by herself. But next, Okuni began to gather other women around her. Many of these women were sex workers. Or they were people who did not fit into society. Okuni trained them to dance. She taught them to sing and act. Soon, these women operated larger shows. The women would put on clothes to look like priests or soldiers. Sometimes they would make fun of these important people. Sometimes they would act out love stories. These plays became very popular. They were funny, bright and often sexual. Other groups began to copy Okuni's style. Soon, people all over Japan were performing kabuki. Okuni retired from performing in 1610. She was like a film star. But her kind of kabuki would not last. Even after it was popular, many kabuki actors were still sex workers. Many people did not like that these workers were performing. They believed it was bad for people watching. And that it would make the men who watched act badly. In 1629, the Japanese government stepped in. They banned women from performing in kabuki. But kabuki did not go away. Instead, men took on Okuni's style. They built theaters. They began writing full plays. Slowly, kabuki turned into a very particular kind of show. Actors would wear bright clothing and they would paint their faces with bright colors. The clothing and face paint would show what kind of character they were playing. A characteristic of kabuki actors is the use of white makeup base for the face. This makes the contrast with the other colors painted on the face more distinct. An evil person would paint themselves to look frightening. A funny character would seem like they were always smiling. Acting in kabuki theater is also very stylized. If the play has a fight, it does not look like a real fight. Instead, the actors will dance. Kabuki also treats emotion differently. Today, many actors will try to show emotion as realistically as they can. They cry when their character is sad. They smile when the character is happy. But in kabuki, actors show strong emotions through their bodies. To do this, they will make poses called mie. Each mie shows what the character is feeling. Some actors will use the same mie in many different plays. Kabuki theater also started using difficult stage tricks. In 1736, kabuki theaters started using moving stages. These stages could go around in a circle or move up and down. Some used trap doors. And in the early 18th century, kabuki theaters began using something called a hanamichi. A hanamichi is a raised bridge that goes from the back of the room to the stage. Many actors would enter over the hanamichi. They would act out scenes above people's heads. Even today, most kabuki theaters have a hanamichi. It is one of the most important parts of kabuki staging. The stories told in kabuki theater also grew more complex. There are many different kinds of kabuki plays. Some are funny. Some are based on Japanese history. Others show religious stories. One of the most famous kabuki plays is called The Treasury of Loyal Retainers. But many English translations call it The 47 Ronan. The 47 Ronan is based on Japanese history. The play begins when an official of the Japanese ruler, or shogun, insults a man named Hagan. Hagan is the lord of a small area of the country. But he is of lower class than the official. In his anger, Hagan attacks the official. Neither man is hurt, but the shogun decides to punish Hagan.  He dismisses all of Hagan's soldiers. And he gives Hagan's land to another lord. Hagan does not believe this punishment is right. He believes the shogun should punish the official too. But Hagan is very loyal. He kills himself with his own sword. But as he dies, he speaks to the head of his soldiers named Yurinosuke. Hagan tells Yurinosuke to make the official pay for his wrongs. The rest of the play follows Yurinosuke and 47 soldiers. These men have no master, but they are loyal to Hagan. They plan to kill the official who wronged him. At the end of the play, they attack the official's home. They find and defeat him. The play shows this as a heroic act. The soldiers did not have to do what their lord asked. But they do what they believe is right. Today, Kabuki Theatre is not as popular as it once was. Most Kabuki plays use very old language. It is difficult for people to understand. Kabuki programs also last for a very long time. Originally, some plays could last for 10 hours. Many theatres do not perform for this long now, but the plays still last for much of the day. Still, Kabuki Theatre continues to change. One Piece and Naruto are important Japanese comics or manga. They are popular all over the world. Some theatres have made Kabuki versions of these manga. Ibizo Ichikawa is a famous Kabuki actor. He has even written a Kabuki version of Star Wars. He spoke to Nippon.com about the changes happening in Kabuki. Of course, it is still important to protect the good old things in our culture. But we cannot let people believe that Kabuki is an art form locked in a box. You cannot view it calmly without making a sound. To change how people understand our art will be very difficult, I think. But it is something that has to be done for Kabuki's sake. Kabuki is important because of its history, but it is also important because it is different. Kabuki offers a particular style of song, dance, and acting with colorful costumes, makeup, and specially built theatrical stages. Kabuki maintains theatrical traditions that combine with new adaptations for modern storytelling. No other kind of performance in the world is like Kabuki. That, more than anything else, is what makes it leave its mark on the world. Does your country have a special kind of theatre? When did it develop? Tell us about it. You can leave a comment on our website or email us at radio at radioenglish.net You can also comment on Facebook at facebook.com. The writer of this program was Dan Christman. The producer was Michio Ozaki. The voices you heard were from the United Kingdom and the United States. All quotes were adapted for this program and voiced by Spotlight. You can listen to this program again and read it on the Internet www.radioenglish.net This program is called Entertainment Around the World, Kabuki Theatre. You can also get our program delivered directly to your Android or Apple device through our free official Spotlight English app. We hope you can join us again for the next Spotlight program. Goodbye!


I'm a bit fearful of running a lightning network wallet on my phone since the app or my phone are not always on. But running a node on my always-on desktop at home is not handy or mobile. Can I run a combination where my phone controls a program on my laptop? Yes, you can. In fact, that's that's what I do. You could do that with your laptop. I generally find it easier and it's not that expensive to buy and run a mini node. So, single board PCs of different types, single board computers, mini computers such as a Raspberry Pi or a small mini PC like an Atom PC with a Celeron processor or something like that plus an external hard drive which you can get a USB hard drive. These things will cost you maybe $150 and for $150 you can run a mini computer that you connect either to your Wi-Fi or to your access point. You just leave it running in the background and you install software that runs and manages a Bitcoin node, a lightning node, and a variety of other nodes. There's a lot of different software packages that do that allow you to run your own node that make it so easy that even non-technical people can launch it and then manage it entirely through a web interface. I would do that. In fact, let me clarify. I do do that and a variety of lightning wallets, Bitcoin wallets and other pieces of software that you run on your smartphone or mobile device will be able to connect directly to your own node and there's instructions on how to do that. It's fairly easy. For example, I run a lightning wallet and all of the infrastructure, the lightning nodes, the Bitcoin node and everything else is on my own mini computer that is sitting next to the access point in the room back there and I access it over a Tor in fact because it uses onion routing so I can access it from anywhere in the world over the Tor network whether I'm home or not remotely and it's my own node and I get the maximum amount of privacy, security, autonomy, self-determination and authoritatively I can look at my Bitcoin and lightning transactions. You could also do this on a Windows machine if you want on an old laptop if you want but let's be honest. Server infrastructure runs better, more securely, more reliably, more consistently with higher availability on Linux. There's a reason why Linux has completely dominated the server market. Linux distributions for Raspberry Pi or for small mini PCs exist that have all of these things ready to run and can do one click installs or maybe just a couple click installs and a very easy to boot and install will do it for you. How can we contribute to mastering the Lightning Network book by providing translations to other languages? I have briefly looked in GitHub but I couldn't find any reference maybe I haven't been booking carefully. Giovanni, I am afraid you cannot yet so the agreement I have with the publisher O'Reilly Media is a bit of a compromise but it's a compromise that pretty much no other publisher makes and this compromise is they allow me to publish it under an open license Creative Commons but they reserve some of the rights for the first year and then after a year if the book is selling and we don't have any problems with distribution they agree to open that license further. So the way we've done it for my previous two books and we're doing it now is in the first year from the moment it's published we issue it under a Creative Commons CC by Creative Commons Attribution NCND non-commercial non-derivative. Now this isn't strictly speaking an open license it is a Creative Commons license you are free to read it you are free to share it you are free to contribute you are free to do many things you are not free to create derivatives meaning mashups and you are not free to republish it on your own commercially or non-commercially because that's a derivative so translations are derivatives so we don't actually start the translation process until a year after publication. Publication is due this year hopefully by the middle of the year we'll have the book published and then the clock starts and in the summer of 2022 it will be open for translation many of the translation off efforts will start earlier but we can't actually use any of them until the license has been changed to CC by SA which is Attribution Share Like which is a fully open license at that point we can publish translations. Thank you for that and I really appreciate your support and your contributions and your efforts in translating by the way if by your name I understand you're Italian if you're interested mastering Ethereum and mastering Bitcoin second edition do not have completed Italian translations as far as I can tell and so if you would like to do that you can look on transifex.com slash antonop and find those books and help those translation efforts on transifex.com slash antonop and there you would be able to translate my existing books which are under open licenses. Is there an upper bound to the lifetime of a lightning channel could I keep it open forever there is no upper bound yes you can keep a lightning channel open forever the protocol implemented in the modern lightning network currently in production does not require you to close channels and that's because of the implementation of a relative time lock called uh check sequence verify which was introduced I believe in 2016 which allowed the lightning network to keep channels open indefinitely the original white paper the lightning paper tells you you can't but that's because that did not exist when the pipe white paper was published you can keep them open forever hi I'm Andreas Antonopoulos if you enjoyed that video consider that it takes a lot of work to produce open and free content that can be shared with everyone around the world this isn't sponsored by some company it's not promoting a product or an altcoin or some kind of investment scheme my goal is simply to help explain the technology of bitcoin and open blockchains to as many people as possible in a neutral way that focuses really on the incredible possibilities that this technology brings us if you'd like to support that mission subscribe to my channel and go on patreon.com slash a antonop where you can participate and help me build better content for more people thank you


So it's time. Let's start again with again a gist of a talk about this time embedded state of the union Please welcome Olivier. Hello everyone So this is a bit of a follow-up to Tim's talk for those who were there an hour ago and But this time I'm going to really focus on what's new for embedded users these cases So first who am I quick introduction? My name is Olivier. Hi. I'm been a G streamer contributor developer for 12 years now At collaboration since 2007 first I started doing video calls on using telepathy for the me go may more platform with Nokia and Since then we've most diversified to basically everything that you can do with G streamer from video editing to embedded systems to transcoding cloud systems But a lot of the work that I've been doing in the last couple years has been on embedded systems Well, this is why I'm going to talk about them today Just a quick introduction what kind of embedded systems use G streamer and the answer is a lot everything that does video Audio and embedded systems you will find G streamer in many many many products there sometimes We're really surprised to find them. So for example in the TV space. It's pretty pretty pretty big On my slide you have some some LG TV some Samsung TVs A lot of these actually have G streamer on the inside on the smart side on the top left You have the Xfinity box from Comcast for those who are not from America Comcast is the biggest cable company in the world and every set of set of box that they ship is Linux box with G streamer for all the playback And then there's a bunch of others bottom left. It's you you view which is a British company more setup box a lot of the TV space has it on the Endpoint but now actually a lot of the recently it's been growing in the in the Production side as they've been transitioning from hardware and FPGAs to software-based workflows and G streamer has seen a lot of traction there Inflate entertainment another big one that people interact with all the time So in a lot of planes almost all the planes modern planes these days by modern I mean from the last 10 to 15 years You go and the end in flight entertainment every time you play a movie. That's G streamer Even in this space station, this is a really cool one. That's one of my favorites. I show it every time. It's a little Camera that floats inside the space station made by the Japanese space agency and that has G streamer inside so I Spoke about these but there's also a bunch of other devices like security cameras A lot of the high-end ones have G streamer on the inside Yes Where So it depends all the new generations are 1.0 generally like the ones that fly in the sky in the planes They're these are probably from 10 years ago. So that's probably whatever version they deployed In the very past, but for example, this is one one stand something. It's pretty recent Smart TVs, they actually some of them are actually quite like this Comcast guys are on 114 as far as I know So there's a lot of these actually they keep up because they want to deploy new features, right? They need the newer dash and the newer HLS features. So they have an incentive to keep up And they have like rapid deployment and all these things in this industry now It's not like especially in the TVs and the cable operator. They really want to deploy new features very quickly So I was saying there's all of these but there's also a bunch of others like all the industrial equipment also that You don't think about but they process video in there and they use G streamer very often because it's on many Embedded chips when you buy them G streamer is the framework that comes already working if you buy exilings FPGA that have a Video encoder there all the framework that xilings enables is G streamer So it's the quickest way to get the working product So I'll give a little summary of the things that we have done that Tim hasn't covered that are really specific to embedded First a lot of work has happened again this year around video for Linux codec support So video for Linux is the Linux kernel API that is used for Things that have a queue So video encoders video decoders amongst other things displaying things capturing from a camera some display devices Some other video processing devices scalars and things like that that are not in the display path That are used in the memory to memory mode So there's a lot of these but a big part is video encoders and decoders in The last year we've merged the HEVC encoder and the coder support we have and jpeg encoder a plug-in and Vi codec, which isn't kind of interesting because it's not a useful codec at all It's only to be able to test the kernel infrastructure So it's a fake codec that is implemented in software in the kernel to be able to test the whole codec Infrastructure inside the kernel without having to deal with the actual hardware Another nice feature that especially on the camera side some cameras took like seconds to probe Because we enumerated a bunch of things that we didn't really need to enumerate so Nikola has done a lot of work and Now the device probing is instant on almost all hardware So you can get the list of all the devices and the relevant capabilities Last year I talked about stable element names for encoders and decoders So originally for video for Linux elements in G streamer We were generating a new element name every time a device would pop up They would connect in the camera and a new element a new camera a new encoder and your fun and the element name would change would appear with a new name and Which means that some on some in a system every time you would reboot it the elements would have different names Which is good, which is okay if you're using like play bin and it's a Auto-generated pipeline because you don't care about the name But if you're doing pipelines by hand it was a bit annoying so now we also we have these still but we also have a set of elements that Have static names and then you give it the device so you can control it more manually So we had this for encoders and decoders last year this year We've also added the same thing but for transformation elements So transformation elements are things like scalers or color converters or different elements that normally convert Raw images into different different formats one thing that His being discussed that's not there yet because the kernels not there is Having good support for stateless codecs. So most codecs that People traditionally use on these embedded hardware. You would just give it the h.264 Bitstream it would give you a decoder stream It did the whole parsing of the bitstream and everything in the hardware side, which was often just firmware running on a different chip But now the new trend is to make the hardware cheaper by moving all of this parsing Onto the CPU side and by doing it in user space and software. So you need slightly different APIs in the kernel There's a lot of work happening with the request API is in the kernel to do exactly that and just remember will support it once once it's there This isn't actually we've been actually working on completely separate subject now We've merged a plugin called IPC pipeline, which I like to talk about because I think it's really cool and it allows you to Split a pipeline into multiple processes But have a single the master pipeline controls all the others So for example the typical case is that you Receive you have one process that talks to a network that download the dash stream from the internet that That is exposed and then it passes the data to another process which actually talks to the hardware decoder So you can separate the hardware decoder the part that and these are often Not as secure as you would like they do a lot of bitstream parsing and things like that so you want to make sure that this part that might be compromised by In that stream is not connected to the network and vice versa So we can actually split the pipeline in multiple levels I have one stage that talks to the network one stage that does the parsing and then separate stage that talks to the hardware and This is also useful to implement the RM sadly but it allows really you to have a Multiple levels of security and it's right now. It's not It's it's you have to create everything manually so it's not like automatic at all right You have to actually create your pipeline manually, but for these kinds of devices You actually want to control exactly what happens in what process and everything Another completely separate thing we've implemented a new mode of interlacing that actually exposes how some of the hardware works So traditionally in interlacing is that you have one frame in Traditional analog TV you have one frame that contains every outline and then the next frame or the next field contains every Evan line so instead of having 30 frames again you have 60 fields and these fields are really half the frame one odd one even odd even and One these come to digital Mostly what you did is that you put both of these fields in the same file in the same frame in the same buffer All right, so you have the odd lines the even lines are are taken at slightly different times and This this is why when you look at it like without properly the interlacing and you'll see a jagged image This is a traditional way to do it some hardware any 265 actually do it in a different way Is that you actually have the front the fields separately in separate buffers at separate times? this allows for a bit lower latency so you can have like You don't have to wait for both fields have been captured to start processing the first one And we've implemented this for GST when X and actually through how the G streamer framework We've done also a bunch of work to reduce the latency in RTP pipelines When we actually try to measure the latency of the actual data We realized that the latency that G streamer claim to have was not really true That there are a bunch of little bugs all around the different elements that actually made the data staying there longer than it should have So we did a bunch of little little bug fixes and now you can actually Push buffers with the latency that it claims to have which it which can be almost zero if you're not doing anything that any queuing We also done some bunch of work in the G streamer open max elements a Lot of this is for the Xilinx Zinc and P platform we fixed a lot of bugs. We've added support for 10-bit video formats for HDR We have a lot more DMA buff and zero copy modes So that you can connect them in different ways and Have one side or the other do the allocation depending on what's better for your specific use case We've had the region of interest. This is a really cool one. I think Basically, it allows the Application to stay in my image this bit is where the really important thing is so put more more bits there when you compress For example, you can have a face recognition algorithm. It says there's a face there So in my video everything else is not that important But the face has to be recognizable or the car plates or things like that or the sub titles If you have subtitles burning to the video, you might want to put more detail there So the text is readable even though you really compress the rest of the video a lot Another little thing is a dynamic frame rate in the encoder. So now the encoder If you change a frame rate it doesn't re-initialize everything it just changes the frame rate so you can change a frame rate really quickly Another nice thing we've done a little bunch of little improvements on the DMA buff Support one of them is that now we do explicit DMA buff synchronization It turns out that the Linux kernel when you use DMA buff first when you access them You actually need to tell it I'm going to access it and now I'm stopping accessing it so that it does the appropriate cache invalidations otherwise you get Corruption, especially on Intel hardware, which is one that actually lots of people use and We have a direct GL DMA buff uploader. So traditionally when you would import a YUV image You would have to use a shader Converted to RGB to be able to be put on the display, but some hardware and particular vivante in the IMX sex They have a they can actually take the YUV and displayed directly Entirely in the hardware so there's a direct uploader now that bypasses this shader that does the conversion so these are the things a bit of a summary of the things I found by looking at the Git log and trying to remember what happened in the last year and These are some things that are actually being worked on more or less actively But I think are kind of interesting There's a bit a lot of activity around neural networks in this tour and your own networks are all the rage this year and A lot of it is to process video to recognize things in video and this room is really good with video so We have for example Nvidia has released something called the deep stream SDK that works both on embedded Tegra and on the cloud side on the and that uses CUDA to do the actual Neural network processing, but all of the video Elements are actually using G streamer This is largely proprietary sadly But the the G streamer actual G streamer is not but all of the interesting bits that Nvidia has done is using their proprietary things But there's also a couple open-source Projects that have been released as one called NN streamer There's something called GSE inspirations meant to be released next month. They promised To do neural networks to G streamer So I expect that in the next year or two will probably have something upstream That everyone can collaborate on to integrate neural network frameworks with G streamer And another thing that's coming up now is different companies are coming up with specialized hardware So instead of using GPUs Yes But we use tensor flow say a lot of these frameworks actually integrate tensor flow or tensor flow RT With G streamer that's like the main one, but I know some people actually using other other frameworks, too Yes, so yes, also specialized hardware is coming I saw a lot of these there are like specialist accelerators that are not like GPUs but that being really designed for AI workloads and these will Require some integration and the streamer is probably going to be a key thing there Another branch that we have not merged in a long time But I have promised to review is Android camera to API so this is a new camera API that Android has had for a couple years now and that is completely More modern than what we're using right now this new API. It's exposed in the NDK so you can use a Native C API to access it instead of going through Java and going back to the C++ It does things like where you can like record a video and take a picture at the same time It allows you to capture multiple streams at the same time like the front camera and the back camera at the same time on some on some phones, etc and to have all the modern features that your Phone app that your camera application as in your phone. They're all exposed for that API. So there is a plug-in in the Git lab that is Meant to be reviewed and hopefully will merge it in the near future There's also a bit a bunch of work about remote tracing So this timer has a tracer framework that allows you to write tracers to trace things and to especially nice to Find performance brought on the next figure out what exactly is going on the pipeline is running and Now there's some interest to do it remotely so to have some infrastructure to forward this information to a separate computer So you can have the tracer running in your embedded device But never nice UI on your computer to know what's going on my pipeline and why are my frames being dropped even though Many indicators say they shouldn't be right. So where's the bottleneck? What is the bug? What have I been doing wrong? another Next step that we're going to do that. We've just just switched to Git lab Where we gain amazing new technologies such as doing a build before you merge the branch and not after and running tests before you merges instead of after so and in this move we've Right now we built for Linux x86 64 we built for Android and we built for Windows And we would really like next to build for an embedded platform To reflect where G-trumor is most used And the next step after that will be to actually run tests on an embedded platform We've built a prototype using a Raspberry Pi and The prototype used Jenkins and a lava so Jenkins to do the build and lava to actually run a device Our goal would be to replace Jenkins with GitLab CI here so that we can actually integrate it nicely in our CI We have a vague plan on how to integrate GitLab CI with lava But I don't know if anyone else has redone it so we actually have to actually see if it if it works to use the GitLab CI APIs to drive lava and Have all the the details together. So This is basically what I had to say You have any questions, yes Yes, you So the Nvidia part is already there it's called deep-stream SDK it works on Jetson and Whatever the server ones are called No, no a point of interest is that as far as I know is only available on Intel and So it needs supports in the you need support in the encoder. I don't know what the NVN API can do So but that then it will be in all platforms Hopefully in few years it will be everywhere. Well, yeah, but especially this region of interesting It really depends on the actual encoder implementation So I don't know what they have in there and their implementation Yeah, go for it Other question I have a tool questions you mentioned before this IPC pipelines Yes, great. There's another plugin called in that pipe Yes, allows to make a separate pipelines and think them together What would be advantages of this one or that one? So the main Thing that IPC pipeline does differently from inter pipe is that IPC pipeline actually controls the pipeline Entirely so when you do you put the the master one in the post state It put all of the other ones in the other processes also in the post state. So the the control is Unified while inter pipeline kind of the point of it is that it's the other way around They're really separate. So you have a standard in the receiver and you can stop Let's say stop the sender and change it and stop the receiver independently Well IPC pipelines really meant to have Things that to the application appears as a single pipeline Even though other bits might actually be separate If I would want to build very dynamic application So if it's like if you want to stop and start different pipes like the inter inter video source sync source and the inter Pipeline, whatever it's called an inter inter pipe inter pipe. Yeah, it's pretty more suitable In IPC pipeline is more meant the trash you meant for like playback cases where you You want to separate things but it's not because you want them to run separately is because you want you want them to actually Appear as as one thing the application And the second question is more on the Embedded side of it, but the same IPC. Yeah, if I'm using like very low-powered IMX6 CPU And I have some hardware source of video if I would split my pipeline in parts How would it pass the buffers between the pipelines? Would it be like a mem copy and it would be yes so moments hit so right now IPC pipeline? It's a it's a socket. So actually copy everything through the kernel For the use case that we're looking at it was fast enough But I had the plan to actually pass Memory buffers using either passing the the file descriptor or shared memory But we never implemented it because it was not required for our use case Last question Question regarding the debugging tools, are you actively working on it or is something already available? So there is something available From the guys from Samsung. I can't remember the name of it is it's open source On this github. Does someone remember the name of the thing that marching worked on? Yes oak tracer oak tracer, yeah, so that's that that's one of the efforts I would like to see it more integrated right now. It's you need to patch things to actually use it live Okay, thank you, Olivier. Thank you


Let's kick off slide one with some audience participation. So, hands up, logo recognition time. Go and hit me. The youngster at the back. It's Mosaic. Yeah, so let me take you back to the very early days of the graphical web. Here we have, yes, the Mosaic browser running probably on Windows 3.1, I guess, 94, not my screenshot. And the nice thing about this is that you can still get to this page if you use the web archive and pull it up on Chrome, and it's still, the beauty of the web writes, everything still works, more or less. So yeah, this is a pull of the Mosaic browser homepage. Early graphical browser, certainly the leading browser on Windows at the time, and X Windows, although it was, as I remember, big and bloaty on X Windows. There were other browsers too, but the thing I really like about this is circa 93, four, the beginning of the graphical web and I bring it up because I feel like with WebAssembly, we're kind of in a similar, maybe not quite so significant, but probably the most significant event since in terms of technology, that's gonna empower the next wave of web applications. One thing I wanna remind us about is that we don't use the word cyber space often enough anymore, so I really kind of missed that. That was such a big thing. I think we should maybe use WebAssembly here, community to bring resurgence of cyberspace. The extra cute thing on here is that back in 93, 94, the Mosaic team, humans kept track of everything that was new on the internet, and every new website that popped up, they noted it and they created a page and they updated this page with what's new on the internet, which was pretty incredible because of course the internet at that time maybe didn't even change day to day. It might be a few days, nothing would change. Of course, not much like that now, every minute. We've got just an insane amount of content creation and content consumption, but back, let me take you back to the 1st of May, 1994, one of my favorites here is that the American Psychological Society has joined both the internet and the web, and they did this because they established both a gopher directory and a WWW, so even in 94, we were a bit skeptical about whether this hyperlinking, gopher was a king of hyperlinking. It was more popular than HTTP, probably until about this point in time. We hit the inflection point and the image tag probably sealed the deal and we were off to the racism. Arrespo Observatory, some really important scientific stuff going on, and up at the top I got a crappy little quiz that I wrote in my off time when I was studying computer science. So I've been doing this now nearly 30 years, and so I was trying to bring my journey of writing crappy web applications right up to date, and I might even have time to do that live in the demo. So yeah, my name is Liam Crilly. I work at EngineX and this talk is about server-side web assembly and mostly about web applications, frankly. So let's get into that. Jumping back to 94 again, and this is my screenshot. I'm so glad I took it, and I still have a table with a date stamp, so I know exactly when I took it. I had my first web app, like everyone else. It was like, come sign my guestbook, and later on I added a counter and we knew how many people have visited three or four or five. But this is HTTP 0.9, straight out of CERN, HTML1, straight out of Tim Berners-Lee, and you could build web applications. But if you know your early HTTP standards, we just have a get. There is no post, there is no file upload. We just got images, we just got fill out forms, but all of these things were ad hoc extensions on these very, very nascent standards, and everyone kind of played catch-up. So Netscape did a thing, Mosaic did a thing. We like that, we don't like that, Blink Tags, et cetera, and the browse of what's happened, and like innovation ran forwards with standards caught up. So how do we build web applications in back in the day? Well, we had no post method and we had no bodies. Well, you could do things like this. This is version two, by the way, of the quiz, and you could have fill out forms and radio buttons and do questionnaires. You could put search indexes on file systems and look for stuff. You could do interesting, useful things. I could write some software and have my mind blown that that software could run on someone else's computer and they didn't have to install any software. Amazing.  So we had query parameters and we just ran them all the way until probably the one K limit. And you could build a surprising amount of stuff this way and stuff got built. But all this stuff just pushed forwards without standardization and there are a lot of mistakes along the way. So things evolved from here and as much as I'm interested in front end development and progression of web apps, I'm gonna focus on the back end side here. If anyone wants to reminisce about Flash, we can do that after the talk. So these first web apps, these things that we're hacking on the end of the query string and passing parameters across and doing useful jobs, we lent on the underlying Unix and POSIX systems. So we sent stuff across. When we got HTTP one, we sent stuff across in standard input. We got stuff back on standard output. We use file descriptors in our applications or whatever, or shell scripts. Whatever we were running. And we sent the HTTP context by which I mean URI input headers when they arrived bodies into the application. That's our context. Then we have a context that's got sent back to the client. The status code, 200 okay, 404, whatever. Again, a bunch of headers and then the response body, that's the thing. And so this HTTP context was made, try to bring this into the application. So developers, people that was hacking around trying to sort of figure out what we could do with this new technology were able to leverage this because the web server would just fork a process. These environment variables and file descriptors were available to the application and we could build stuff. And that took a certain way, but it didn't scale. And we had things like VAR-CGI come along and we were able to manage processes a bit more smartly. And then 96, 97, we got all the components that the LAMP stack came around. So PHP, MySQL thing came around that time. The LAMP stack just was an amazing marketing wrapper around these technologies, but it gave people huge confidence that you could choose these tools, get support, use open source, and this powered really the next wave of web applications. It powered the first generation of Facebook. So LAMP stack took us way forwards and because PHP had this HTTP context available to the developer as a principle object, like a primitive in-cycle language, with your types and data structures that you could do. I wanna get ahead of, easy. So that was tremendous and it had its own challenges. And then the next wave comes around when web native languages come along. So things like Python, Ruby, or Ruby on Rails in particular, made the next kind of leap forward. And every time we do this, we give developers more choice and we provide this HTTP context in a more and more native and natural way to developers. Backend frameworks, like yes, I mentioned Ruby on Rails, one of my personal favorites, Flask for Python. And then the coexistence of the back end framework and the front end framework to provide a really rich front end user experience that wasn't completely re-implementing what's happening in the back end. That sort of brings us up to date. The latest cloud native languages such as Golang and Node.js kind of take this little one step further because what they do is they bring a web server with them.  Node.js and Golang have these things, they're just an inherent part of that stack. And that's good and bad. So when you deploy those applications, Golang or Node.js in production, for example, you put a reverse proxy in front of them because you kind of need to guarantee a bit of scale. You usually need to do a few things at the HTTP layer that the application layer shouldn't or doesn't want to do. TLS, who wants to build certificates into the application code? People talk about doing end-to-end TLS. You go to the reverse proxy and the application does the rest of it. So what I'm advocating for is decoupling the network layer that handles HTTP and the runtime. So that's courses for courses. What happened in these earlier stages is that we allowed the web technology, the network technology to go at its own pace, innovative at its own pace and the application runtime to do the same. And they were both able to go on their own journeys and be extended and innovative in their own direction. And everything kept working. Like you had, you followed your Apache lifecycle and got more and more features. End-to-Dex comes along, does the same thing. Tomcat, IES. The application code, and the languages, are able to do their own thing as well. And as soon as we bring the two things together, we stall that innovation. Not to mention separation of concerns and security boundaries and giving people choices and independent control over changing each part of the stack. So, come back to this point. Because what do developers really need? They want to be able to write their business logic, write the application code. They tend to care absolutely nothing about the bottom of the stack. Certainly not the network layer. And they don't want to be introduced to it either. They might be happy understanding what an HTTP request looks like in HTTP One and what a response looks like. And that's the input and output into the system. So, here we have a very simple, let's call it a JSON REST microservice. If you'll indulge me. That takes an array of numbers and adds them together. And someone has to deal with the floating point rounding error. And so this is what the application code might look like. This is what a good developer experience looks like in this case. And I'll use Ruby again as an example. Getting access to the HTTP request body is a very simple statement. Sending back an HTTP status code. It's sort of an implicit part of the response. It's not even, there's not even a declaration about what this is. It's just, okay, this is the end. Though it must be the status code. This must be HTTP. And it allows and allows developers to build really interesting applications. And only have to deal with a little messy bit in the middle where the business logic is. This is an absolute slam dunk for WebAssembly, right? We've got this request response patterned HTTP. We've got nearly, like we've got 30-ish years of HTTP infrastructure, tooling, experience, middle boxes, proxies, caching engines, cloud services. All this stuff is ready to deliver applications at scale. We've got proven at hyperscale with the world we live in today. And we've got WebAssembly, which once bites in and bites out, okay? HTTP request, HTTP response. This is just too tempting. We've got to build some really interesting things here. And so the group that I work in, the NGINX saw this and thought, well, you just have to do this. This is gonna not only be an amazing plugin model for extending like the NGINX proxy that we already have, but it's gonna be able to change what things look like at the runtime. We just need network layer, TLS layer, HTTP parsing, request routing, and code execution. Code execution is the only thing we really care about here. Everything else is a solved problem. So let's separate these concerns. Let's minimize the application code to the sandbox, the WebAssembly runtime, because we've got pretty good solutions for handling lower down the stack. And we know the developers don't care about lower down the stack. So this kind of became a plan. So let me, what we have to do right is just do some magic in the middle and get this HTTP context in and out. So as I mentioned, right, I've worked at NGINX for a number of years. I take it there's some people who know NGINX in the room. Anyone using NGINX today in their more or less daily lives or have used it in the past? Yeah, awesome. Anyone used NGINX unit or heard of it before? We do a great job of keeping this a secret. This is a six year old project that was created by the original creator of NGINX. And after that, he wrote a JavaScript engine for NGINX and after that, he wrote units. And because it's from the original creator of NGINX, like all the things that he wish he'd done differently in the first version had been sitting on the shelf as prototype code and we kind of, a few things came together at the birth of this project. So yeah, it's an HTTP engine. It's a web server, it can serve static files, but the main difference between the NGINX project most people already know is that it is really supposed to be an application runtime. It's veriting application code. And so we've gone through this period from the lamp stack onwards where we tend to have a web server reverse proxy, something that serves a static files, something that does the dynamic content. And Node.js is a great example. The reason you put a reverse proxy in front of Node.js, not because Node.js is bad or is slow, it's really great, but you probably wanna serve your static files in a simpler way. And so reverse proxy sits there, routes to Node.js, serve static files. The sort of mission of unit was to simplify that stack. So we wanna be able to basically more tightly configure the web server to know about the application code rather than hand it off separately. But we didn't wanna tightly couple it, so it's a multi-process system.  So it's got Nebuchlayer, it's got TLS layer, it's got HTTP parsing, it's actually a little bit faster than EngineX HTTP parsing, and it's got a really nice request routing layer. And then it's got a shared memory model where it manages the processes of the application code, in this case, Python. So we talk in the language of the HTTP context for the programming language. So in Python's case, we have WSGI, we have ASGI for the async model, depending on what you're building. PHP has SAPI, there's a number of different languages, each with their own interface description for how the URI and the headers get across and how they get back again. And so it's felt like just, this is gonna be easy. This is not our first rodeo, we've done this at PHP, Python, Ruby, Perl, Go, no, Java. We've got all these backend web frameworks that we know work really well. And all we're gonna do is we're gonna drop, we're gonna basically do some copy paste code and we're gonna drop WebAssembly in and it's gonna be great. So this is the design, we've already got this system, it's got HTTP context, we've got this nice fast shared memory system which passes across to the application runtime. All we need to do from the HTTP context, which is now in another process, is initialize wasm runtime. So yeah, let's use wasm time, let's have it load the module and we'll have our app process, basically tell wasm time what to do, initialize the memory and we'll just copy the context in, business logic can run and we'll send it all back again. Now we chose wasm time, A, because it's nice and mature, B, because it was written in Rust and we didn't have a Rust experience for Unity. So we always like to add more languages and the more we can cover, the easier it is for people to have a good experience with multiple languages. And by the way, you can run these multiple languages on the same host or in the same container because it's one process managed at the top and it manages it on the other piece. And because it has nice C bindings. So unit is written in C, the main engine projects written in C, we've got lots of really good C developers and that's the go-to choice. So we set off about three or four months ago to execute on this plan and no one else really knew anything about web assembly and we kind of scraped our way through and found out where things are happening and how to make it all work. And it was tough. And so I'm speaking for the team. I'm not a C developer. I'm on the, I've seen a lot, right? But I'm on the product management side of the house and I just watched the team through this process. So we started where you would start in the simplest possible way. We built a kind of CGI. So this is the initial sort of forking model and using POSIX standard to sort of, WASM time has lots of WASI stuff built in so we could do the standard in, standard out trick. And we knew it was gonna be throw away because we know it doesn't scale. So our application process launched WASM time. We call WASM time memory grown, allocates some space for the HTTP context to come in and that seemed to work more or less. And then we started writing our first WASM modules to test in this environment. And because we've got C developers, they wanted to write it in C and then compile the C to WASM. So, okay, it's not where I'd start, but yeah, go with your tool chain of choice. And that also kind of worked until anything beyond the most simple, small request would just take full. So things fell over. We had memory allocation happening in our app process calling WASM time through the WASM time memory grow. And then we had the WASM module allocating memory inside its own linear memory space and that they were just bang into each other and collide and we had lots of, lots of problems. So, turns out all of this was mostly avoidable because not many people are gonna be writing WASM modules in C. It's not, the benefit's not really that great. And the, turns out the C tool chain is one of the least mature WASM tool chains, but we didn't know this yet.  We had to look at the, this is the, at the top of the slide here is the default WASM C linker stack and heaps layout, memory layout for the WebAssembly runtime. And the data was very small and sorry, the stack was very small. And so we took these three steps to finally get some reliability in our ability to write C modules that compile to WASM. So we just memory layout to the bottom set. So we separated the stack and the heap and we made the stack size a lot larger. And then we stopped trying to allocate memory from afar. So we just let WASM time initialize and then within the WASM module itself, it asks the memory that it needs. And what we're doing today is that we're basically allocating 32 megabytes and all of the HTTP context in and out goes across this 32 meg, unless you need more. And we do this in a streaming fashion so you can have like multi gigabyte file uploads to a WASM module and you can do it all through this 32 megabyte interface. Unless you want, you can write that out of the desk for example without needing any more memory. So we know this is a model we use elsewhere, sort of buffering trick in NJX to avoid large memory growth. And this seems to be a model we can also apply. But to get this working reliably, this turns out to be the set of linker flags that we need to compile C code to a WASM module for our runtime interface. And so as you can imagine, this isn't the first iteration of that flag. We now have a library that hides all this for you. So like everybody else, we've had to write an SDK to make the developer experience close to approachable. Okay, so going back to really what developers need, we want a really easy approachable way of just focusing on the business logic and having access to the HTTP context in and out in a really easy way. And this is our, nearly fits on a slide, right? And this is our best effort today. So we have a Rust SDK as well. And I know that everyone that's tried to do a self-signed web assembly has had to build an SDK because the standards aren't there yet. But unlike 94, 95, 96, like the whole community is working together to get that stuff right, and we're not gonna have this sort of bifurcated sort of a Wild West standards force. So it's not super pretty, but we don't have any unsafe blocks in our Rust code. We can now write Rust and this is, of course, this is the simplest sort of hollow world. We have access to bodies not shown here. So I'll ask you to squint at this and imagine it's an AI inference model. This works. And I think I'm gonna push my luck and see if I've got enough time to do a little demo here too. So let's try this. Here's some Rust code. So I'll take you back to the HTTP example that we saw earlier of adding numbers together. And let's take a look at this. This is quite old code, so I do still have some unsafe. So let me get through the initialization phase and scroll down here, right? So the allocate 4K for all the headers, which is usually four AKs, usually the limit web servers anyway. We've got a dependency on this JSON provider. Here we get the context of the request, request body, and then we do some actual Rust and iterate over this array and the numbers together. So again, imagine this is an AI model instead. When we just hack some raw JSON back and send content type header. So there's a complete web app, you know, six-yard lines of Rust, and let's run it. So I've got unit installed. As you can see, I got this from HomeBrew. Sorry, yeah, I did, yeah. So room install unit, unit wasm, gets you everything you need here. So every runtime is a plug-in module, so it's not one big, bloaty runtime. So I'm gonna start it up. I'll attach it to the console so we can see what it's doing. And we'll see all the, if I have a problem, we'll see the logs on the output. Oh, yeah, I meant attach it to the console. Okay, so it starts, multi-processes start, look at all my available runtimes I can use, so we're just gonna use whereas I'm here. Is 0.1, is a tech preview. And right, I've got a controller for config, I've got the main process that handles all the processes and I've got the router process. So that's good, let's build this Rust. So it's a library, we'll call it add. Let's copy that Rust file into lib.rs. And now, let's add the dependencies. So, unit wasm is a dependency, we have a crate for that which is influence the SDK, so we don't have to worry about any of that. And I needed that JSON library. And I think I'm ready to build. And I can't remember the wazzy32, maybe build will help me. Yes, so half of these are just to get JSON. Done, so let's have a look at target. Wazzy, I didn't do anything smart, so I got. I forgot a thing. So, this is a library, not a console application. So, we decided that we need a dynamic library because it's gonna be loaded by Wazzy32. And so that is, all right, I just gotta cheat add this to cargo.toml. So, I need to say that this crate type is a dynamic library type. And now I can build it again. And now I should find a .wazm file where I expect to find a target wazzy debug. There it is, all right, so add.wazm. Just because that's a null, I can't type all that. I'm gonna copy it down here. And just to prove it, all right, tell you what, let me just get rid of that. I have one .wazm file, and that's what we're gonna use. So, unit started, I'm gonna use, it's configured with a JSON REST interface. So, there's no config files apart from state. So, I'm gonna use the unit C command line tool to see what, it's listening on a unit socket, which is why it's a bit of a painful to use curl. So, here's the current configuration. We've got listeners, routes, and applications. Let's edit this. So, I'm gonna listen on port 9000. I can do all sorts of things. And what am I gonna do when I get there? Pass it to the routes. So, I only got three objects to manage, listeners, routes, and applications. As you see, routes is an array, so we get things in order. And I'm just gonna put like a default action here just so I can make sure it's working properly. Love typing JSON. That should work. Every request to this web server is just a mutual for, but anyone that can pass JSON in real time is welcome to help me. Tell you what, I'll do one thing at a time. Yay. Okay, that's now their action. I like that. Let's check it. So, let's get the wazm running, and I'm not gonna type this because it's a little bit gnarly. I've got a app.json. So, I gotta define an application called add. Type is wazm, so that knows what module we're gonna use. And then the module itself, I sort of have to, we have language modules and wazm modules, and I can't do anything about the ambiguity, so, sorry. So we specified the module itself, right? So I ran this from the current directory that I'm in, so I'm just gonna skip the whole path thing. And then these are all of our SDK helpers that allows for memory allocation to happen safely inside the wazm module, and how to initialize and end the handler itself. So I'm just gonna pass that straight to the application's object, so I can reference any of these configurations directly. And that's done, and in the log output standard error, we can see that we started a prototype, which makes spawning new processes fast, and the application itself. And I've hit PS, I've now got, so unit is now managing the application processes. There's a bunch of other config, I can say scale to 20, scale down to five, and it will auto scale the rest of it. One last thing to do, I need to attach a route to this application. So config routes currently is just doing 204 for everything, so let's do a new add one more object, and this time we're gonna have to match on something, so match on anything. Let's match on a URI of add, yes. And what happens when that matches is the action. Pass this request to the application's object, that's add, we just created it. So that should stitch that together. I can match on headers or methods, I can say it has to be a post, right? But that should be enough. Still can't do JSON, let's try it again. Match on URI of add, and maybe that was it. Let's see the whole config. All right, listening on 9000, all requests go to routes, routes will match on slash add or fall through to the default. So I can still, let's get my, I still send some garbage URI, I'll still get 204, but let's now construct something a bit more useful. So I'll call slash add and nice thing about this HTTP tool is that I can construct JSON without typing it. As we know, that's challenging. So let's add some numbers together. So we sent this object and some wasm ran and gave us an answer. And there's like, there's no other magic here. There's nothing that's happening. There's only the add to wasm file and there's only the processes that can just manage. So the developer experience for this allows for Rust developers fairly easy access to the environment that they know well, tool chain they know well, and we literally just need to drop the one file in. And this really is the why we're excited about wasm in general and server-side wasm. So I've now got that wasm file and I wanna be able to push that anywhere. I wanna run that in your cloud, my cloud, someone else's server. I'm gonna take that and self-host, run it through a test framework, and I've got one build pipeline. And that's, as well as all the other amazing advantages of WebAssembly around performance and security, this build once, run anywhere, promise is, like that's the real deal. So as I mentioned, developer experience is king and this only works if the developer experience is great. It only works if we have universal portability across all of the server-side runtimes. So we can't wait to throw away that SDK we just spent months building because Wazzy HTTP and the Wazzy cloud world are the things that we really need for this stuff to take off. So this is the beginning of our journey. We're really excited to see what's already planned for Wazzy HTTP and Wazzy Preview 2. And we start work on that next week. And if that's true, we're in a position to have the same sort of burst of innovation and separation of concerns that allows these Wazm files to go anywhere for the utility of Wazm and the future of server-side web applications to take to the next, yeah, web apps to the next level. So that's it. I hope that's interesting. I'll leave you some notes about the NGNX unit project. If you're wondering, well, what about NGNX proper? That's why I came. This is the learning we'll be taking into that project to allow for extending NGNX with Wazm as a plug-in model. Feel free to, I think I've gone over time. So you can grab me after, grab me at the booth or drop me a note. Thanks.