e I think if you wanted to build your own like hardware or like try out like if you have some vendor that you use at work that you think is cool or something definitely finding that overlap is great I think for me like I think just finding these like mircats was great right because I didn't necessarily want to go through like you know which motherboard is compatible I didn't have I don't have any IPMI at home and so that is downside but I think that's definitely you know for me at least I think that's definitely a next step is like automating that and then getting mass to like automatically power on and off boxes is super awesome so yeah so I'd love to see your blog post your talk next year what you decide all right great talk next year is there's a great talk there any considerations for security in your lab or is it is it walled off so that no one can really get to it whoof I am so thankful for all my colleagues who actually know how security works I'm very very stoked I think I like like for me I was like oh yeah like there's certs like cert off this is great that's all I need I think I'm definitely I'm actually starting to look to my colleagues at Slack built something called Nebula which is like alternative to like wire guard or tail scale so I think something like that so you don't necessarily need to like expose ports on the open internet is really really appealing right now naively I am I have opened the like task server port on the router so I do have that like exposed so yeah I think like you know that is something that I'm definitely thinking about so I think something like tail scale if you haven't looked into it it seems super cool so but that basically creates like a like a very like interesting virtual private network so you don't actually need to like make this connection and you don't need a ton of extra servers you basically have this like network where only your servers can communicate with each other no matter where they are in the world it's like it feels like magic it's super cool so so Hey Joe so did you have any problems with UEFI on the mirror cats or were that was that pretty straightforward from like configuring it to do the the netboot fortunately I did not have many problems I think I just had like I think was more like a personal problem where I was just like oh my god it's not bios anymore like when did this happen like I don't so I think that was just something that like I needed to like correct my misunderstanding so yeah I think that just like appreciating that I think helped a lot because then I think like it helped me not be afraid of like going to the firmware anymore if that makes sense like I think just like feeling more comfortable like knowing what's going on like knowing how to partition things like knowing which disks I'm dealing with I think having that for those fundamentals I think was actually super valuable but yeah I think I think overall the UFI was was pretty solid so and then the lemur pro is rocking core boot which is awesome so yeah open source firmware very sweet any other questions you're making me run nope nope my ip is 127 001 I think same as yours I think all right what do you do for electrical power protection for your equipment great question I grabbed some I don't even know if it's actually good or not but I did have the same question and I found something on Amazon for like 70 bucks or whatever and it's not a battery backup I don't really care about a power supply or sorry I don't care about a battery backup just yet but it is something that it's like it seems like just a fancy surge protector so I am like I didn't necessarily want to like plug stuff straight into the wall just in case like there's some like dirty power coming in or something so it should I think I forget the details of that but it should like even out the curve power curve or something is that a thing that thing yeah that sounds that sounds reasonable cool cool cool thanks everybody yeah I appreciate it yeah I would definitely look into that if it sounds like you're thinking about that already so okay so more about your presentation how did you make the slide so I really liked your theme there yeah that's the project is called look at me I cannot take any credit for that I should have added it back here but right there the first bullet point right there is the GitHub repo so it's a Python project so if you just pip install that that'll that'll work yeah I was like blown away it's it's pretty awesome so yeah you just write markdown and like each markdown document is a new slide so it's super cool so for a home network Kubernetes cluster how small and portable can you go there are there I missed the beginning on the hardware you're running but can you just can it be as small as Raspberry Pis or smaller yeah and also can the can the cluster be based on a wireless network sure yeah out of the box yeah so I think you can definitely set up like your servers on a wireless network I think this is where like me being a little annoying I think I just wanted to have that you know there's nothing better than just plugging in you can totally do wireless and it's probably going to be fine and then I mean like if the if you could UEFI boot over wireless if it wouldn't that be cool do we have any experts that actually know the answer to this question Logan yeah I'm hearing not yet probably coming soon I don't much I mean like the time scales we're talking about like soon as relative before scale 30 before scale 30 that might be a thing yeah I don't think we'll get like WPA 7 support though but you know maybe we'll see so but yeah there's a ton of youtube videos like if you just search for like home lab there's a ton of like awesome folks who are like super stoked about this this one vlogger Techno Tim just started this thing called 100 Days of Home Lab which is awesome wow this is wild so I think this is just a few months ago actually so there's a ton of folks who are like talking about their home labs like making a ton of videos kind of like really building and encouraging a lot of this stuff so there's like infinite ways to set this up so I also want to give a huge plug to the self-hosted podcast just great folks if you're not subscribed to like the destination Linux podcast like the tux digital network like a lot of those folks are doing awesome work and like really like encouraging a lot of people to do cool stuff with technology so there's just like a ton of like momentum that's being built to this community which is super cool yeah and I think somebody made like a 40 raspberry pie like cube cluster or something that they were talking about on youtube which is cool yeah that's awesome all right give it up again for Joe rock and roll thank you so much thanks everybody all right next up on the cloud native track and I believe a half hour we have Nathaniel McCallum for Weirkan the container bootloader saga oh there is a break test test test test test check two one two check check check check check check all right test three one two three check test test test check check check check all right um gonna check check check okay here I am I'm gonna read the the schedule oh no oh okay check check check check oh okay so we've got some I think uh tweets here check check okay just gonna read some of these uh tweets my favorite part of my talk at oh check check check check check one two three perhaps the most oh sometimes the questions of people who don't contribute as much is can we get them to do more instead we should dot dot dot rather than asking people to do oh congratulations team GitLab won the Soquel Linux Expo most most interactive booth hoard great copy I'm still talking to the mic check check check last day at scale 19 sometimes the question of people who don't contribute as much is can we get them to do more instead we should okay my favorite part of my talk at scale 19 today in an hour in the marina room is a personal note that says q dot dot dot perhaps the most important sentence in our Spake's entire talk testing testing no not surprised at all this is God you need to make some changes to your life okay I've got uh 130 so we're going to go ahead and start we've saved the best session for last here at the cloud native track of scale and we've got Nathaniel McCallum here to talk about we're canned which is a cool mechanism for booting containers into a computer potentially even including an os image so with that said I'll let you take it away Nathaniel great thank you very much how's everybody doing today last last day of the conference has it been good yeah what's your favorite talk we'll just take a take three people favorite talk so far don't say this one because you're cheating you like the keynote yeah home lab pod man also great talk Dan Walsh is always a delight former co-worker of mine at red hat and spent many years with him he's a great guy so I need to start off this this talk with a bit of a warning this presentation does contain gratuitous fantasy and Dungeons and Dragons references so you have been warned and with that we're just gonna we're gonna jump right in so we this is this talk is called working the bootloader container saga and we're gonna start here with with chapter one hopefully all of your character sheets have been prepared and you are ready to play a round of Dungeons and Dragons we're gonna start off by identifying our hero our hero is actually not the project in this talk it is the project called Enarchs Enarchs is an open source project which runs web assembly workloads in trusted execution environments does anybody does everybody know what trusted execution environments are good I am glad nobody knows except for a few over here so the idea of a trusted execution environment is that there's a mode of the CPU which encrypts all of the pages of the memory so for a particular application or a subset of that application and then can do hardware based attestation so we can use Enarchs to bring up an application in an environment that's actually encrypted while it's running and we can prove to others that that environment is tamper proof and encrypted and then deliver keys and such into that which is a great way to write micro services because you don't actually have to rely upon the security of the underlying operating system just the the trusted execution environment platform itself Enarchs is also a Linux foundation project within the Confidential Computing Consortium in fact we were the inaugural project for that division within the Linux foundation so let me give you a little bit of background at Enarchs history Enarchs started in 2019 and it started basically by us going to Intel and AMD to explore their new technologies Intel SGX and AMD SEV the first commit I think was in February of 2019 something like that and it was actually written in Austin, Texas while we were talking with the developers there and basically the idea was how can we make this portable you have technologies that are on different platforms they work in very, very different ways how can we create trusted execution environments so they're the same across all hardware platforms and in 2019 we decided late 2019 we decided to focus on WebAssembly and we merged WebAssembly support in 2020 late 2020 and kernel development at this point is happening in earnest so you have lots of patches for these new hardware technologies coming into the Linux kernel and then in 2021 SGX the first for version one lands in the Linux kernel version two is I'm very excited about this is about to land in the next kernel release just now and SEV the first version of that began to land in the Linux kernel in 2021 and in mid 2021 my company Profian which is a startup we launched to provide commercial support to NRX and to allow you to be able to deploy applications in the cloud and on the edge in ways that actually don't depend upon the security of the underlying operating system and of course in 2022 we're going to be releasing Profian Assure which is our enterprise support for for NRX but this is let me give you a brief demo of what and how NRX works so you can do this on your own there's no special magic here and what I'm about to show you you do need the latest generation of hardware in order to take advantage of the hardware capabilities so this is ice lake Xeon for Intel or the Milan generation for AMD and since not everybody has these servers sitting in their back pocket we decided to make a website called try.nrx.dev and you can click start now and we'll randomly pick one of the platforms we ended up with SGX in this case and on the right hand side of the screen we have a small echo server this is written in Rust using the Tokyo IO framework and we're the only thing that's web assembly specific in this whole application is this line because we're going to create or we're going to use a socket that was already set up by the runtime and other than that this is all just native code so we're going to listen for new connections which we're going to accept when a new connection comes in we're going to spawn a background task in Tokyo and we're going to handle its output so we're going to read whatever the input is from that socket and we're going to handle a disconnection and then we're going to write it back so a very simple standard echo server and we're going to build this for web assembly so normally we would just do a cargo build but since it's web assembly we're going to specify the target wasm32wazzy and now our application is built for oh I think somebody's playing with the server they just took it down let me try the let me try the other one nope that's how you know it's a live demo anyway we could have uploaded this to the server and we could have shown it running on try.enarchs.dev if it was not down so maybe I'll come back to this at the end of the presentation see if it's back up maybe somebody's deploying a new patch or something okay so that's actually not what this talk is about this talk is actually about working and so one of the challenging things that we had with Enarchs is that we need to move on now to chapter number two testing your metal and yes I know this is a pun I hope I didn't injure anyone testing your metal is of course spelt M-E-T-T-L-E but we actually the problem we have is that we need their metal because when we're developing this software and we want to test it we want to test it rigorously because this is security software and so building Enarchs needs bare metal we test to development kernels we test hardware cooperating user space demons we do application development and we do it on heterogeneous hardware in heterogeneous clouds so all of the clouds basically are offering bare metal in various forms these days but how you actually boot an operating system across all of these different clouds is incredibly different and they're not really the same and we had other problems as well but this is a this is really a many headed dragon as you can see team at here and so we have a variety of different ways that are in use for booting bare metal you have the standard sort of pixie system which is you use DHCP it tells you the TFTP place to get your kernel in an RD it fetches those over the network and boots them that's one way another way is IPixie this is commonly used by equinex for example and so IPixie is like pixie but it's scriptable so you can basically write some scripts to do things and it also has like HTTPS support and stuff so it's not not just TFTP but it sort of has TLS support because IPixie only supports really old Cypher suites so if you actually try to download images from say you know github or pretty much any modern web server it's going to fail because it can actually do the TLS negotiation you also have the ISO method some places will allow you to upload an ISO and to boot that which is great usually you'll have to go through an operating system install although there's ways you can sort of make that automated then you have the EFI direct boot route which is where you put an EFI image on an EFI partition and that gets loaded directly then you have the operating system image this is usually a derivation of the ISO format or some other thing there's a a three letter cloud for example that requires you to have a custom image format for their servers and then lastly we had to mix this also with stuff on prem and so being able to just do like a usb mass storage boot for example is another way we may want to boot these things and so because we have all of these servers doing rigorous testing of of all of the hardware and we have all of these clouds and we have all of these different ways to boot things we really needed a system that could sort of wrangle all of these together so we really need to tackle that that multi-headed dragon and we have specific goals first of all we want our process to be iterative and building a whole operating system image is yes it's iterative but the cycle is just really long to be able to build a whole OS image we want to do something that was faster more lightweight second was we wanted something that was stateless we didn't want to install an operating system and then have to manage the state of that operating system over a long period of time we wanted to boot into an OS where everything came up with the with the bare metal and then when we rebooted it was exactly the same state as when it started before third was we wanted everything to be declarative so we wanted a clear recipe to be able to build our operating system images and we but we also test on multiple different operating systems and so while for example we could use something like kick start to do an install on fedora or red hat and other systems provide other automated systems as well that would have required us to maintain a bunch of different methods for composing these images across multiple operating systems fourth is we want the process to be reproducible this is incredibly important for any development or testing environment particularly when you're dealing with a very rapid development across changing situations you need to always be able to reproduce something at a particular point in time so we wanted to be able to reproduce every single OS build the fifth thing is that we wanted to be homogeneous so in other words if we're booting an operating system it shouldn't matter which hardware we're booting it on and it shouldn't matter which cloud we're booting on the entire process should be the same and further the development procedure for building that operating and booting that operating system should be the same for everyone across the line lastly we wanted it to be infrastructure-less and now I know that there's not really there's not really such a thing as no infrastructure but basically there's a lot of infrastructure that's provided for free from a lot of different places and we didn't want to have to set up infrastructure to manage our infrastructure easy and lightweight particularly as we had a small development team we didn't want to have to start doing IT ops immediately and so we simply needed more agility we needed the ability to move faster and to create environments that were reproducible and so forth so let's move on to chapter three Arming the party we already had as I mentioned some resources available to us so we start with Git right I mean if you haven't heard of Git yet you should probably go learn about it immediately Git is pretty much the de facto way in our industry these days to manage iterative changes and so we wanted to be able to use Git to manage the control of producing images that we could use for for booting and we wanted the whole thing to be declarative right so the image is produced in a declarative way from the recipe that is in Git and this gives us those two things we really wanted which is rapid iteration and change control second we do have available CICD resources there are many many examples of this all over the web I listed two here we have GitHub actions where you can run some code on their server in the cloud on your GitHub pushes or the same with GitLab for CI you know if you tag something if you do a push of code or whatever you can run code on all of those events and so this is essentially some free compute resources for us and we wanted to make sure that we could do this efficiently and lots of people are doing this today right this is a really well worn path committing a Docker file for example and building a container in GitHub is not exactly the bleeding edge of technology the last thing is that we have available container repos and so basically forges like GitHub and GitLab both have the ability to start with that Docker file that's in GitHub do the build of the Docker image and then push it into a container repo and now it's actually staged for deployment and so we didn't want to introduce additional infrastructure on top of this we just wanted to say like look we've got our whole operating system packaged as a container in that container repo I just want to boot the dang thing how can I get it to boot and so there's there's an easy part of this and there's a hard part of this the easy part of this is actually distributing the kernel and in in at RD because if you look at a container image it's basically a set of layers it's a set of tar balls that are layered and contains all the changes you've made to a container over time and we can stuff things in this and then we can distribute this right typically you don't distribute a kernel and in it ramfs in a container because it's not going to use that kernel or for anything and it would bloat the size of the container so but so we needed a way to do that and that's really easy actually you just kind of install the distros kernel package that's easy step out of the box there's a few other little tweaks you have to do but it's it's not very difficult the second part was the harder part and that's container aware boot loader so we needed something that could start up on the system no matter what boot process we had right whether we were doing iPixie whether we're doing native boot whether we're booting off an ISO or whatever we wanted we needed some bit of code there that could recognize how to get a container image and then figure out a way to actually boot that kernel and so we came up with the idea to make containers more metal and we started a project called working working is available on GitLab so gitlab.com slash working slash working and it's written in rust and I believe it's under the Apache 2 license and this is the bit of code that does that harder part in the early part of the the bootstrapping process working provides these four things these are the main things there's a few other things it does but these are the main four things that it does first is it is it does container muxing it basically takes a bunch of different container layers from a container repository starts downloading all of those it downloads them in order as they would be actually laid out on the file system when the layering was collapsed and we stream this whole process so we're actually un tearing them as they're downloaded and then we stream it into it in in our D image so we actually convert the docker format into an in it or D format one of the other things we need to do as part of this process is we need to actually extract the kernel from this image because although we can boot the inner MFS we need to get the kernel out separately to be able to handle that so that when we do the container muxing basically we're just grabbing the kernel from the container image as well as converting the container image into an inner MFS the second thing that we that it provides is K exec booting how many how many people are familiar with K exec here? Okay, we've got a few so K exec it was designed as a recovery mode for the Linux kernel basically the idea is that you can load one you've got your Linux kernel up and running and you can load a backup Linux kernel into memory and it's sitting there waiting to go and when your primary kernel crashes it can jump into the secondary kernel and now you can be able to do debugging or keep the system running and so K exec is an interesting tool that we can actually use so working is both a binary but it's also a that binary plus a very small operating system which includes in a Linux kernel and so first we boot the work in Linux kernel that's in the in the work in image we download the container do all of that stuff prepare the inner MFS and then we do a K exec pivot into the new kernel so you actually as part of your booting you're going to boot the Linux kernel twice the first time for the bootloader and then the second time from the kernel that's been extracted from from the container the third thing that work and provides is config persistence so we did have some way we did need to have some way to persist some information about that particular deployment so when you're actually deploying a container on bare metal hardware for example some clouds have fairly sophisticated networking setups maybe DHCP doesn't work at all and you have to set up bonding and static IPs that are associated with that particular server now because we have a goal for working to be stateless we can't just install an operating system and then configure it all so we needed some way to be able to persist that configuration across reboots and so the way that we do this is working takes that local state and persists in EFI variables so basically the first time you boot the system you're going to have to set it up for whatever it's networking whatever the needs are tell it which container to boot once you have it all working then you save that config into EFI and from that point on that server is just going to boot automatically the last thing that it does is network setup and I've already hinted to this on different clouds you have different networking scenarios some will just have it working with DHCP some will use IPv6 slack others will just have static configuration you're expected to set up things like bonding and your IP addresses manually and so work and also provides the ability to take that networking setup from EFI and then actually set up that network inside the the work and as it boots and so this is what it actually looks like to prepare a bootable container if you've ever used a Docker file that's exactly what this is basically the top just says we're going to take in an argument which is called tag and this is just so that we can iteratively build this on multiple releases of Debian and we're going to say from Debian so this is we're just going to choose to to base this on top of the the core Debian image and then we're going to do the next step which is to install the packages we need in order to get it working so the first thing we're going to do is we're going to update the metadata with apt and then we're going to upgrade all of the existing packages that are there so now we have the freshest base Reaper then we're going to install in-it-remfs tools and the reason for this is actually surprising the reason is is that if we don't install in-it-remfs tools it will be installed as a dependency of system D and the kernel which we're going to install next and the problem is we actually don't want to to generate in-it-remfs because the whole image the whole container image is going to be the in-it-remfs and so that can easily double that if you're building a small container for example generating in-it-remfs can easily double the size of the container image so all we're doing here is we're installing in-it-remfs tools and then we're linking over the in-it-remfs process with bin true and what this will do is that whenever the system tries to generate an in-it-remfs it will just immediately succeed and now we don't actually generate an in-it-remfs so lastly we install the kernel Linux image generic followed by systemd and now we have all of the packages available the last line by the way just clears the apt cache so that we don't store that in the container makes the container image smaller so now we have all the packages installed that we need to make a bootable container we need to do two additional things the first is that we need to sim link the kernel that we've just installed into the special file name working dot kernel and this is just so that working knows where to find the image you may have multiple kernel images installed or so forth and we didn't want to have to like do magical interpretation so you make the decision you decide what kernel you want to boot and just link it into the right place for us the last thing that you need to do is you need to actually set up systemd to run when the system starts and you just do that by by simlinking the binary to slash in it and that's it we've created a bootable container image that's that's really all it took this is the an example for debian but we have I think five or six different os's examples on the git lab site so you can go and check them out there all right so now we're prepared we've gotten our party we're armed for battle and the question is are we going to fight the battle yes we're going to fight the battle so we're going to do the battle for boot and this is what it would look like to actually boot this thing the first two lines here curl we're all we're doing is downloading the pre-built work in kernel and in not rd so all of this is is built in a reproducible way on git lab as you can see exactly what goes into those images there's no no surprises the last thing we're going to do is we're going to do direct boot so if you mentioned if if you remember before I talked about direct boot is one of the options this is where you basically load the kernel and interface directly and that's what we're doing with just we're using queemoo for this so we're going to launch queemoo and we're going to come back to the kernel command line in a moment but go to the second option which is the kernel and in order to you just see that all we've done is specify the files that we downloaded and that's the that's the work and boot loader the next three lines are just basic setup we're going to enable kvm and we're going to say we don't want any graphics for this particular image and that's because we're just going to dump all of the stuff to the terminal and record it that way and then the last thing is just to set we want four gigs of memory for this vm so that's that's pretty straightforward now let's go back to the kernel command line we can provide a command line here but think about it this way if working is an onion there's an outer kernel and there's an inner kernel and so you need to have the ability to pass arguments to the outer kernel which is the work and boot loader but you also need to be able to pass arguments to the inner kernel which may be different from the outer kernel perhaps the outer kernel on the inner kernel have different hardware support because they're different versions and maybe the the software behaves differently so all we do in this example the first thing we do is we just tell it which console to use which is the serial console and that's why we disable the no graphic well you can see all of the work and argument start with Wyr so Wyr dat dash arg this is passing log level equals three to the inner kernel all right so anything that you specify is just a normal kernel command line argument here will be passed to the outer kernel all the work and arguments all the things that start with Wyr are going to apply to the inner kernel the last one here is Wyr.image which is just the the slug to the the container that we want to boot and that's pretty much it and I'll actually give you a demo of this so first we're going to open up this this shell script and all this is is just a convenient way to put all the same stuff in there so you see all the same arguments formatted in a slightly different way and you can see we have the the command line as well as the kernel in an rd there and we're just going to run this Q image and so we're now booting the outer kernel in working that's what's happening right now it's it's being done silently and as soon as that kernel boots and the networking comes up we are going to start okay we've already started when now that we're going to deploy this container we're downloading that container right now where as I said stream converting it into an inner mfs and then finally we are booting it and so now we are actually booting the inner kernel which you can see here and that's it we've done we've done the whole thing so but there's other forms of booting as well so this is an exact same example the only difference here is that we're setting up ovmf as our efi and in qemu and we're booting an iso image instead in this case we don't have access to set the kernel command line directly so now we're looking to boot at a a little simple boot loader here to give us a menu to type command line stuff so this is what would be like if it was on bare hardware and you're booting off an iso image maybe from a usb key or whatever and so we're going to say there's the working argument again specify the console we're going to specify the image which is that debian image and the last thing we're going to do is we're going to save these parameters using the efi equals write command to efi and so now that efi image that we enabled with qemu when it was being booted has now persisted all of those options that we we typed and so on the next boot we don't even have to type anything that we did the in the first case because everything has been persisted to efi other than that it's exactly the same experience as before we're downloading our container image we are stream converting it into an nana ram fs and I do have a note here there's some very very old very obscure hardware that if you attempt to write to efi it bricks it and I just wanted to warn people of that fact but it's very unlikely you'll actually hit that anywhere in production and so we've written all of the variables now and we're now booting a second time so this is the second time of our boot the only reason I'm editing command line image here is because we have to put stuff over serial console but all of the other variables are now being pulled from from the efi and we'll see it again we scanned meaning that we found we found that information from efi we've loaded our kernel and now we're booting the inter kernel and that's it now keep in mind the entire operating system is memory resident now doesn't that cost you a lot of memory not really the reason for this is even if you are deploying a large operating system if you keep your container image small yes it's all going to be memory a resident in memory however if you're just keeping your container image the stuff that you use all of that stuff is going to get loaded from disk into memory anyway and the operating system is going to cash it so even if you installed that same exact container image to disk and then booted it normally you're still going to get roughly the same amount of stuff in memory plus or minus so it's it's not actually super painful I'll give you a quick overview of the work and boot arguments as we saw before weird.img tells you which container you want to boot weird.arg is any arguments that you want to pass to the inner kernel and that argument can be specified as many times as you have arguments to pass to the inner kernel the next one is weird.net and this one takes a little bit of explaining so there's a kind option and then a file section and key and ignore the kind option for a moment the file section and the key are essentially the structure of an any file so if you're familiar with what an any file looks like that's exactly what this is it's the name of a file there's a section in the file and then there is a key and a value pair in any file and that's because surprise surprise we're generating an any file using using this argument and the purpose for this is that it's actually generating the network D configuration so if you saw before we installed system D system D comes bundled with system D network D which has a full list of networking operations you can do very sophisticated networking setups with network D and so there's really the sky is the limit and this is a very very repeatable pattern basically if you say that you know if you put these arguments and it's just going to generate the network D configuration file during its boot and then network D will start up whatever that networking is when it's booted so we don't even have to document that for you as long as you can understand that those things transition to an any file just go read the network D man page and it will tell you more information than you ever wanted to know about networking the last two boot arguments are the EFI write and EFI clear options and these allow you to persist whatever changes you've specified to EFI or conversely clear them if you've said done something that's wrong and you can't get it to boot and you're just stuck in this boot loop you can specify the clear option it will clear out all of those options and then you can try again from scratch so the most important thing about working is that it is also itself a bootable container the only difference is that working is a pre-generated kernel internet ram fs so where normally you boot them on the fly the kernel internet ram fs that you downloaded are just generated ahead of time and that's the only difference and so what you're actually seeing here is that we are chain booting work-ins so in other words we're booting into a one version of work-in and then we're telling it go boot this other container where the other container is itself a work-in image so right now we've loaded the work-in itself as a second boot loader and we've now chain booted and so this can be useful for example if you were to write work-in into a ram for example where you could then use work-in to chain boot onto a network downloaded work-in and now you can have updates of the boot loader itself as you're actually doing your boot and so yeah that's it now we've we've chain booted work-in now you may be wondering after all of this what it feels like this is what it feels like to use work-in and anybody's Stranger Things fans yeah all right so that's that's my talk any any questions if you've got some hold up your hands so we can have the audience hear it it might be people will so how do you track whether or not you've already written EFI to individual devices isn't there some opportunity to actually mess things up there I'm not quite understanding the question the question was how do you keep track of whether you've written EFI to the devices or not you can always overwrite anything you specify as a boot parameter is going to override what is actually an EFI so there's always a recovery mode in terms of whether or not you've written before it sort of doesn't matter because you could just rewrite again and it is sort of the intent of booting work-in and if you haven't noticed from all the metal puns we're doing this for bare metal right and so the intent here is that you're going to have to be setting up a bare metal system even as good as the clouds are today and provisioning bare metal there is some manual process presumed to be involved we want that process to be as minimal as possible and then to rapidly iterate from there on out so hopefully that answers the question any anybody else do you support sorry that was not um do you support Cali Linux for this I haven't tried you're welcome to try it's a very simple Dockerfile recipe so if there's Cali Linux recipe you know Dockerfiles for doing containers with Cali Linux then install a kernel and make sure you have the right sim links in place and it should work but is Cali Linux system debased yes okay anything that system debased is going to be a really easy experience it starts getting much harder when you get outside of that realm because then you have to do things like modify handcrafted scripts to do that has certain assumptions about things but anything that system D is going to work very well anybody else okay I've I've got a few I was just curious about using this for the use case of handling edge devices that are unintended and wondering if this could be a mechanism to install updateable OS images out there at these edge locations absolutely and in fact it would be a great companion to use with NRX which is the thing we talked about at the beginning NRX if you if you think about it this way it draws in the security profile of your application so that when you're deploying that application on the edge which has a lower security domain than a traditional one you want that security domain to be as tight as possible around the application so you write and deploy your application using NRX and then you could deploy the operating system on which it runs using working and that's that's a great pair you might ask about the security of the host operating system in that case well it mostly doesn't matter that's the purpose of confidential computing the idea is to install essentially what is an insecure operating system and then have a secure trusted execution environment within which your app runs and the only thing that the operating system provides is resources resource management so how much memory is the application using right is it balanced effectively with other applications that are running and so forth but because it's running in a trusted execution environment the the kernel can't see even in ring zero you can't see into the memory of that application because it's all guaranteed by the hardware so yes it is actually a really nice setup for particularly those purposes follow-up then aligned with this trusted execution environment is there any value on that device of spending extra money to put a TPM module or some other module on the physical hardware or does this take care of the same kinds of problems anything you can do with a TPM you can also do with a trusted execution environment and you can do better with a trusted execution environment with a TPM you basically just get limited key management as well as secure boot secure boot is a great marketing phrase and what it means is that you when you turn on the operating system all of the components of the operating system are measured throughout the boot process and at the end the result of that you can perform cryptographic operations using that measurement in the TPM in the case of confidential computing the security of that chain doesn't really matter so much or at least not as much as traditionally it was required to and so you can deploy applications in a way that's a very efficient they have a much smaller much tighter security boundary any key provisioning for example doesn't need to go into the TPM because the application itself is encrypted while it's running so you can do an attestation to a remote party you can get in cryptographic keys or generate them internally so that no human ever sees them and get the public key signed by another party with attestation and then your application can run in this way that's provisioned but it's a very tight hardened security boundary somebody over here had a question but I'm still sort of struggling with containers in general overall and I'm a security person essentially what we're looking at here is just nested containers right I mean this is no that there's there's no actual container involved with this we're using the container format as a way to package and deliver an operating system that is downloaded over the network that's network boot booted essentially and so it allows us to reuse all of our development infrastructure for preparing our operating system images when it downloads the container image it's just a set of tar balls and then we unpack those tar balls in a very careful way so that it efficiently creates an init ramfs and then we boot that init ramfs so once it's booted there's no more container involved there's no container runtime there's no namespaces there's no container at all you just have an operating system but we use the container packaging format in order to efficiently deliver it okay that's cool so if the operating system if you're trying to actually boot something like again edge use case if you're trying to actually boot something that's effectively going to run a single application okay how much of the sort of a knit tooling do I actually need so I think it's if you're just if you're just trying to boot a single application yeah since you're like the the actual the use case that I have is is it's actually a device that collects data for sensors sure if you want to run your own in it process on that device and have a single process there's nothing that stops you from doing so so you could you could completely eliminate the system D step remember when we installed system D you could install the kernel but you could leave system D completely off of off of that you could install your own binary as in it and now it's running as PID 1 and that's all you have right you don't need any other infrastructure you probably do need some resources though for like when it goes down and when it crashes that's where system D starts to become very helpful but if you but if you have you know a hardware watchdog that can determine if the kernels crashed and reboot the system then yeah go out go go to to your heart's content we've got a couple more in the back but I'm sure I think we still have time okay follow up to his device access this might be more of an nRx question but at that edge use case is it okay for this inner OS to get to IO devices storage devices etc so if you're using nRx to actually protect your application then your application is encrypted at runtime all of its memory pages there's no way to tamper and look inside in fact if you go to nRx.dev slash crypto we have a hack challenge right now we're basically challenging anyone to see if they can hack the nRx system and get that data that's inside we yeah once you've basically built and deployed your application with nRx that's a very tight package right and just your application is encrypted using a separate key that key is governed by the hardware all of the memory is encrypted or decrypted in the CPU directly by the memory controller and the operating system has no access to any of that your application then runs nRx provides transparent encryption for all of your network operations provides transparent encryption or at least will in a future release for all of your disk operations and so once your application is in the nRx deployment system there is it never comes out unencrypted unless you intentionally do so and so basically all of the secret material you're used to trying to have to protect in the operating system level you don't have to try to protect that because you can't actually read into that memory inside of the nRx keep at all so the we can we can de escalate the necessity of having a really tight like secure boot image which is really really hard to provision I don't know if you've noticed we've had we've had TPM devices going on oh it's at least 15 years at this point and they're still poorly integrated into our systems and the reason they are is because it's an incredibly fragile system and one chain breaks the entirety of the system in fact I've even filed a bug there's a situation in Fedora where if you install if you install on a Windows device with secure boot the fact that it reordered the boot process breaks the encrypted drive in Windows right that's just how fragile the systems are when you when you have to depend upon secure boot for this and so what we're trying to do is we're trying to create a much better system where you can have full general compute available with the same level of hardware protections that you would expect out of a TPM talking specifically about working yeah you mentioned edge you can't say edge without arm sure is there any arm support on the working side there is no reason it won't support arm all the major arm systems these days are EFI everything's built using Rust which compiles find to arm I haven't actually built it and tested it but I would absolutely welcome contributions for supporting arm so I was wondering about the kernel do you have the kernel config like what's the is it like a standard kernel it's Debian it's Debian Debian skirmish okay yeah so there's no modifications on top of it yeah if you if you can be if you can boot the Debian kernel you can boot working okay last call any other questions I just went to the GitLab project I didn't see any documentation or anything on there is there anywhere where we can look up the documentation on there's pretty extensive documentation in the in the readme so if you go to oh maybe I just didn't look at the read in the right place and just scroll down there's there's quite a bit of documentation okay I think I didn't see that because I'm on my phone yeah no worries all right thank you yeah I I have the same problem too anybody else does this work with pixie or ipixie yes yeah there's basically you have the work in produced kernel and innet remf s you can boot those any way you want we also produce from that an iso image that just boots those that kernel and innet remf s so whether you use the kernel and innet remf s to boot or whether you use the iso image to boot you can configure it any way you want on your hardware and it'll it'll work so using pixie for example pixie just fetches the kernel and innet remf s from tftp over the network so just put the work in kernel and innet remf s on your tftp tftp server and you there you go it's booted any others okay well I want to thank Nathaniel this has been a great talk thanks for having me let's let's give them a round of applause safe journey home everybody okay this is the Canadian network of oh no is it no this is how how does that go there's this great wonderful pun but I can't get it rattled out oh yes this is the dominion network of the Canadian broad corp incestration check that does too check sounds good enjoy can you hi everybody we're still a few minutes from getting started but feel free to grab a seat feel free to grab some ice cream or some coffee or some snacks from the back of the room as a sweet way to end scale so if you are in the back of the room again there are plenty of seats come come come grab a spot even socially distant we are good and again enjoy some ice cream in the back of the room we'll get started promptly at three o'clock which is just a few minutes away so give away for some of the activities we did it this is the convention in Maryland yes all the parts it was a kind of a snap clock that he proposed for the moment yeah and I went to the room it had only gone out on the internet at the time Joe yeah it said I stood up and said is anyone technically yeah cool thanks for making it out Rick cool good afternoon everybody I know we're all super excited for the presentation in just a moment so I'd ask if we can start to start to settle down start to grab some seats there is not not many sessions that will will pack a room at four o'clock on a sorry three o'clock on a Sunday I know we're all super excited about this one so thanks for coming out again at the risk of you all running out of your seats one last time which I don't encourage there is still some ice cream in the back of the room for those of you that are looking for some there is some there is a okay maybe it's gone there is definitely some coffee and some other things again thank you to our amazing break sponsors with with that that that supported the event you know VMware matter most and linking this is what happens after four days of being here I will I will get that in a second and I will thank them hopefully they're not watching the recording data bricks yes sorry data bricks as well so thank you again so we are super excited to have everybody back here together after that two and a half year break for those of you that here we're here in the morning it's maybe closer to a four or five hour break but the thanks again we've got a few administrative things and then we'll get we'll go ahead and get started with our our main attraction with Dr. Sir so how many of you participated in the expo floor passport program slash scavenger hunt how many you feel like you got all the answers right okay wash less hand much fewer hands turns out only one did and he was tenacious in doing so he was tracking me down asking me for suggestions tips trying to figure out if I knew the answers I did not because I didn't put together the program and it looked hard so I didn't personally do it I'm also don't I also don't think I'm allowed to give myself a prize so we've actually got I see him in the room here so Kyle do you want to come join me on stage and tell us a little bit about what they have one and then I will tell you who it is everybody so what we have for the raffle winner today is a fully decked out leave room 14 laptop what makes it special so one of the things so when we made it we looked into what are all the things that we learned from all the past laptops we made that we think would make this our dream laptops so in addition to having hardware kill switches that shut off the kill the the a webcam and microphone and the Wi-Fi and Bluetooth it has it runs core boot firmware it also runs in it of open source embedded controller firmware as well that we that we developed and in general it's just like the best locked up that where we've been able to make so far so thank you very much Kyle so I'll give you one fun fact on Kyle Kyle is the only scale speaker that I know of in the last 20 years that we've been running the conference to have a literal explosion occur in their talk please I know it might sound hilarious but please don't leave cans of Coke directly behind the hot fans on projectors in the session rooms I'm sure nobody intended to do it but it made for a fun surprise for for for Kyle and a sticky situation for the hotel so I'll stop keeping you all wondering if you won the one person who did win is Dr. Sam Coleman in the back of the room come on up and for those of you that know Sam you've been coming to scale since what 2004 2004 2004 and you were at the time you helped us run the education track among other things of our call correctly the time or two the time or two open source software and education and so at the time I don't think you were Dr. Coleman at the time I think you were just Sam but congratulations I hope you enjoy this free software powered laptop open from again courtesy of the purism team and Kyle and before you take off today we'll want to do a quick picture for the purism folks as a thank you for that awesome so again for those of thank you for coming out for the closing keynote this is the first time we're doing this but we thought it was an important opportunity given we had we had what's had to do an auspicious speaker joining us and so mind just the last few things again we will be back here well not here but in Pasadena a few miles away next year March 9th through the 12th returning to our returning to our home after all of this COVID fun I appreciate you all sticking with us through the masks through the vaccine mandates through the the tests and everything else I know it's not fun I hope that by next year we can skip a lot of this no promises it's not pandemics are not in my control but I do know that we will be back together all things all things being being cool so with that in mind our next speaker doesn't I don't think needs an introduction but I first heard of him in high school I was reading a book called where the wizards stay up late the history of the internet if you've not read it it's a fantastic overview of just all of the early days of the internet how things came together at various at various institutions to build what we now most of us run our careers on today everything from how this conference was organized to what we did through the pandemic to how I get cat litter through the mail all organized and coordinated over to CPIP I don't know if Dr. Surf had that in mind when he did it but that was interesting so so yeah we're super excited the other thing that the interesting piece here is Dr. Surf did this while he was at UCLA and for those of you that don't know when scale started 20 years ago that was done by students at UCLA and USC and UCSB and CSUN and a few other local schools when we were all just but we we children in our freshman and sophomore years of college thinking that running conferences would be easy one thing I can tell you it is is that after 20 years the only thing I have learned is that it is not easy but but yes so super excited to have Dr. Surf and one of the things that has been a pleasure for me over the last 20 years is just getting to meet my my internet and sort of CS heroes as we bring them as we bring them on stage here for keynotes and for other activities and so without further ado I give you I give you Dr. Surf is joining us all the way from Virginia and appreciate appreciate him making the time as well as and for joining us so thank you Dr. Surf thank you all very much you know when people clap before you said anything my first reaction is to sit down because it won't get any better than that now you all clearly understand the tactic that's been employed here you wait until the last session of the four days of ruling fantastic I looked at the at the list of sessions holy moly you guys have got an enormous amount of useful information and I'm sure in the corridors in between lots of other very useful ideas have been flying back and forth so what you do on the last days you get the talking dinosaur to come out and and it doesn't matter what the dinosaur says the fact that he can still talk is amazing so I was I think I have been given a few suggestions though about things to talk about and I understand that when you get to be my age you tell anecdotes and so so we're gonna go looking to the past and look at a few anecdotes and I see more than a few gray hairs in the audience which means yeah I don't have any left but my beard is doing okay so for some of you this will be I hope a kind of a fun reminiscence of things past that have taken us to where we are today and where we are today is a big challenge and I'll try to close my talk with some thoughts about things that we might collectively do to meet some of those challenges and what I find most appealing I think about you in particular is that you've been doing this conference now for 20 years and there is this fellow feeling of collaborative sense of responsibility that you bring to the table that I wish everyone who writes software would bring to the table so we're going to come to to that in the last few thoughts of this talk but let me start out by jumping into the time machine for a second and take you back to 1969 at UCLA not very far away there were four nodes by the end of December 1969 in operation at these four locations which were selected by the Defense Advanced Research Projects Agency for particular reasons UCLA was selected as the first node because Len Kleinrock who is still a professor at UCLA ran the Network Measurement Center and my job as a graduate student there was to write software for the Sigma 7 machine to measure the performance of the ARPANET and compare with the queuing theoretic models that Len Kleinrock students were developing so we could compare what the predictions were from the queuing theory models and see what actually came out from the measurements and I can tell you that the queuing theoretic models were always beautiful and mathematical and pristine and everything else but they didn't always predict what actually happened in a real network and I suspected every one of you knows exactly why that's the case so UCLA is Network Measurement Center then SRI International was the second node and that was because Doug Engelbart was at the Augmentation Research Center at SRI International his belief and the belief of JCR Licklider who was running the Information Processing Technique Selfless at DARPA believed that computers could be a way of augmenting our capabilities and I would say that they were right our capabilities today what we do every single day are often augmented by the kinds of software that you write and other people write to help us do things that we couldn't do on our own not least of which for example was searching the entire world wide web so that was the second node and then third one was UCSB because they were doing some really interesting work on the presentation of complex functions on a screen so you could see what the computations were producing and finally University of Utah because they were very much involved in computer graphics at the time now you can imagine computer graphics in 1969 are not like what they are today but one of the guys at Utah invented one of the first hidden line removal algorithms so that you were doing 3D rendering and you wanted to show what it looked like what the surface looked like he had to remove the parts that couldn't be seen so he found it was John Mornock whom some of you will know started a famous company called Adobe some years later so that's 1969 many years ago and this is a picture that was taken in 1994 it was the 25th anniversary of the ARPANET on the left you see John Postel who sadly passed away in 1998 in the middle of Steve Crocker who led the network working group that did all of the host-to-host protocols the telnet protocols the FTP and eventually SMTP for email so he was the leader of that group and he was at UCLA as was John Postel all three of us went to the same high school Van Nuys High in the San Fernando Valley I don't know must have been something in the air but but we became good friends at UCLA as graduate students it took us all day to organize this shoot for Newsweek Magazine because we had to draw all those pictures with the backdrop hanging there then we had to go find the zucchinis and the yellow squash and the five five pound chins of coffee and then string it all together now this is Newsweek Magazine 1994 and we thought we would put up a geek joke for those who understood it so if you look carefully at this network you notice that it's mouth to mouth and ear to ear but there's no mouth to ear this network would never work so that was our little geek joke in 1994 but we were celebrating the that anniversary of the ARPANET because without it we would not have even bothered to do the internet so now I'm going to skip very very fast forward here past the original early designs of TCP and then later GCP IP this was a demonstration that I called for after I moved to Virginia in 1976 to run the internet program for the defense department and I wanted very very much to show that the TCP IP protocols could really work across different kinds of networks over considerable distances so we had a mobile packet radio network in the San Francisco Bay Area driving up up and down 101 radiating packets like crazy through a gateway which had been configured to send those packets all the way across the ARPANET which by that time extended into Europe over an internal satellite hop from ETAM West Virginia to Tonham in Sweden and then a landline to Norway to the Norwegian Defense Research Establishment and another landline down to University College London and then popped out of the ARPANET at that point into another gateway that led to the Atlantic Packet satellite net which was based on Idlesap 4a hanging over the Atlantic with multiple ground stations all vying for access to one single satellite channel so it was like having an Ethernet channel in the sky and then through the packet satellite network to ETAM West Virginia down to another ground station back into the ARPANET all the way across the US again to USC Information Sciences Institute so if you do the math of course going from the Roving Vehicle in San Francisco down to Marina del Rey is about 400 miles but the packets had gone through two synchronous satellite hops back and forth so it's about 100,000 miles and I remember when we did it and it worked I was jumping around in my office saying it works it works you know like it couldn't possibly have worked listen if it's you know you know that if it works if software works it's a miracle so so so that was a really important demonstration from my point of view as the program manager for this now we'll skip forward now we're in the early and mid 1980s after the ARPANET and the and those three network demonstration was done and further standardization was done then we implemented the TCP IP protocols on every operating system we could get our hands on in 1982 John Postel announced that we are going to switch over from the NCP host to host protocols of the ARPANET to the TCP IP protocols of the internet on January 1, 1983 so this is January of 82 we make this announcement and everybody sort of wumbles a bit you know you know everything seems to be working okay with the host to host protocols and filnet and so on so why do we have to do that and you say well you know if you don't do that I won't fund your program next year oh oh okay I get that so I had Dan Lynch whom some of you know was the founder of Interop was measuring how many implementations of TCP IP he could detect on the ARPANET and so he would report to me you know once every couple of weeks and I could see the you know the career was going up and then somewhere around the summertime it flattens out so you know I'm a bureaucrat at this point so what do you do well you clearly create incentives so I called the defense communications agency it's now called DISA and I said will you shut off the capability of the ARPANET to carry anything except TCP IP they could do that turns out there was a way to do that and I said turn it off for a day so they did of course the phone bringing off the hook what's the matter with you you blankity blankity blank you know can't can't get any email you know files nowhere and I said I just want you to know I can do that so you know oh okay so curve starts going up again then somewhere around late summer it's you know October or something it flattens out again so I called DCA turned it off for two days phone rings off the hook everybody made it on January 1, 1983 except for two guys and they pleaded you know for some kind of mercy and we said okay we'll give you another month or two so everybody made it up now how many everybody's were there there were only 400 computers on the internet at that point 400 as opposed to 400 million or whatever the numbers are today if you include the IoT devices and the mobiles we're into the multiple billions so now so it's now 1983 NSF is starting to invest in internet technology for hooking computer science departments up to the ARPANET and then to the NSF network backbone and they quickly run out of gas they essentially sent out an RFP asking for a higher speed network now remember the backbone speeds of the ARPANET were 50 kilobits a second okay that was that was broadband in 1969 so NSFnet launches with 50 kilobits and and basically runs out of gas it switches to 1.5 megabits and lasts for a little while and then it switches to higher speeds and we just kept going you know eventually we end up in the multi gigabit range so NSF makes this big investment they build a backbone network they build about a dozen intermediate level networks to serve about 3,000 universities the purpose for which is to help those universities get access to the five supercomputer centers that NSF and the Department of Energy are investing in so the NSFnet comes up and the supercomputer centers come up and not long thereafter NASA and the Department of Energy say this looks like a good thing so DOE builds the ESnet backbone and NASA builds the NSI or NASA science internet backbone so during the mid to late 1980s we're starting to see substantial implementation of course if we go to look at the internet today and now some of you will know I'm cheating right because this picture was actually generated from the BGP backbone back in 1999 but it was so colorful that I thought I would keep it because it lets me illustrate something that you know and that is that the internet is very dispersed and there are literally tens of thousands if not more networks running each operator picks hardware and software to run decides who to connect to and on what terms and conditions there were no central decisions about your business model who you connected with the only thing we asked of everybody is please run the same protocols so that you could interoperate we wanted to encourage Bob Kahn and I to encourage people to build pieces of internet find someone to connect to and let the system grow so that's what it looks like today except bigger and more colorful it is true that there is only one centralization element and that is trying to coordinate IP address allocation and domain name assignment so that they're unique and that's pretty important but ICANN doesn't dictate what is done with those things it just tries to make sure that if you want a domain name that it's assigned uniquely to one organization or an IP address block to an autonomous system so this is just my little memory of milestones I'm sure that every one of you who's been engaged in this game have your own favorite milestones so by no means should you take this as the definitive list of important milestones they're just the ones that I happen to remember one very important one comes after the internet goes operational in 93 and Cisco systems figures out that they can make money by selling routers to universities that want to get hooked up to the net and put local area networks in place oh by the way I did leave out something important 1973 Bob Kahn comes to visit me in my office at Stanford and says we have a problem what do you mean we and he says and he says well the ARPANET worked and now the defense department wants to see if it can use computers in command and control but he right away realizes if you want to do that the computers are going to be in mobile vehicles and ships at sea and airplanes and the ARPANET was built out of dedicated telephone lines connecting everything together well you can't connect the ships together that way because they get all tangled up the tanks run over the tanks run over the wires and they break in the airplanes never making off the ground so he was already starting to work on the packet radio net and the packet satellite net at the time that we met so we started working on the TCP design about a mile or two away Bob McCaff and David Boggs are busy inventing Ethernet at Xerox part so that was the fourth packet switching technology that was born during this 1970s early 1970s period so Cisco is the first to start commercializing this stuff the way we used to build routers was to find a computer and a graduate student and you wrap the graduate student around the computer and turn it into a router problem with that is we were running out of graduate students so Cisco figures that out also in 84 it becomes very clear that this thing is scaling up and that we can't go keep sending the host.txt file around to map the domain names into IP addresses so we needed something that was more scalable so Paul Maka Petrus and John Postel invented DNS and it evolves over time but it certainly has scaled dramatically it's a it's a pretty brilliant piece of work so around 1988 by this time Dan Lynch is doing Interop and it's up in the San Francisco Bay Area and I walk in in 1988 he started in 1986 this small thing it was mostly lectures but then it was an exhibit oh and the deal was you can't exhibit unless you show that you can interwork with everybody else so they bring out the show net this big fat yellow coaxial cable you all had to plug into it and then show that you could talk to everybody else in the show so Eric Benamo who is then the CEO of 3Com the company that Bob McHalf started to sell ethernet and I walked into the show and the first thing we encounter is a two-story Cisco thing you know display and so I turned to Eric and I said Eric how much do those displays cost and he says about $250,000 this is 1988 and I'm sort of sitting here and saying that's a lot of money and and he said that doesn't count the people who had to stay there and man the booth for a week and so I'm just standing there and my jaw is dropping thinking somebody thinks they can make money out of the internet that's amazing so so at at this point I'm starting to wonder because you know how are we going to let the rest of the the general population get access to this thing because up until that time all the networks were funded by the government agencies NASA, DISA or how to well yeah DISA wasn't also NASA DISA, NSF, DOE and DARPA so there was an appropriate use policy that said no commercial traffic will flow on a government sponsored backbone and you could imagine a rationale for that they didn't want government resources to support commercial activity but it became increasingly clear that even the people who were doing research under government grant actually needed access to commercial services that could be reached on the internet if the if it were permitted so at this point I'm working with Bob Khan at his company the Corporation for National Research Initiatives and I had taken a little break from the internet from roughly 82 to 86 to build something called MCI mail which is a commercial email service so in 88 I'm sitting here thinking okay so what could I say to the federal government that would let me break the appropriate use policy without really appearing to want to do that so I called the federal networking council which at the time was mostly NASA DARPA NSF and I leave out NASA DARPA NSF and DOE their program managers were the people who managed the internet policy so I called them and I said would it be okay if I tried to connect the MCI mail system to the internet to see if we could get the email stuff to interwork and to my surprise they said well okay you know you can mail for a year and this is just an experiment so by summer or so of 1989 we announced we have a gateway between the MCI mail system and the internet and as soon as we make that announcement all of the other commercial email service providers which are islands into themselves you can't talk to anybody unless you all have account so things like telemail and on time compu-serve yes all of those say wait a minute these MCI guys can't have this special treatment we want on too and so the government says okay so they all get hooked up and two things happen the first thing that happens is they all discover that all of their customers who used to be trapped in a wall of garden can talk to all the other customers of their competitors plus everybody on the internet because they were all compatible through the internet mail protocols and that was a little surprise and then later mail becomes almost free so much for that business model and but the second thing that happens about the same time maybe a little later in the years three commercial internet services pop up because we've just broken the AUP limitation so UUNet in Virginia and PSINet in Virginia and Surfnet in San Diego all get started in 1989 okay yeah this is anecdote number no whatever 17 Surfnet used to be spelled S-U-R-F-Net and of course you do that you're in San Diego what else would you do so so they had a whole campaign all laid out t-shirts you know surf the internet and all that stuff and then a couple weeks before they actually launch somebody discovers that there's an organization in the Netherlands called Surfnet which is you know it's a Dutch acronym and they are building a network to connect the universities in the Netherlands so they can't call themselves F-C-U-R-F-Net so they're Susan Ostrada was the executive director at the time and so somebody says well why don't we change our name to the California Educational Research Foundation Network because you know it sounds the same and then somebody says maybe we should call Vint so they call me up and they say can we call it Surfnet and my first reaction was you know if they screw this up am I going to be embarrassed and I thought about it it's more wait a minute now people name their kids after other people and if the kids don't come out right they don't blame the people they name them so I said sure so I flew out to California in 1989 and we Susan and I had one of the plastic bottle full of glitter and we smashed it on a Cisco router and we launched the Surfnet so by that time we're starting to see real commercial services pop up Rick Adams was the founder of UUNET and so we're talking 1989 so in 1997 he sold the company for two billion dollars to metropolitan fiber systems which on the same day was acquired by Worldcom for 14 billion dollars so he made out okay that worked that was good so just you know picking a few more of these things the big deal after commercialization of course is when Tim Rensley announces the World Wide Web which is late December as I recall and I don't think too many people actually noticed he was doing it on a next machine which was very cool at CERN but not too many people noticed and except for these two guys Mark Andreessen and Eric Bina at the National Center for Supercomputer Applications in Urbana-Champaign and they look at the text-based interface and they say boy it would be really cool if we could make a more graphical interface wouldn't it so they do Mosaic comes out around 1993 everybody notices because suddenly the internet looks like a magazine with formatted text and imagery and eventually streaming audio and video so that was a big deal and Jim Clark the founder of Silicon Graphics takes one look at the Mosaic browser and he says this is a big deal remember he'd started Silicon Graphics which had turned out to be based on another chip that ARPA funded called the Geometry Engine so he flies out and he takes Mark Andreessen and Eric Bina maybe a few other people back to the West Coast to start Netscape Communications in 1994 and by that time I had left Bob Conn's organization to rejoin MCI to put them in the internet business and the first thing they wanted to do is build the MCI mall okay so I fly out to Netscape and buy 7 million dollars worth of licenses for Netscape's browser and server and the first thing I asked them to do is to figure out how to avoid having my servers filled with partial transactions that won't ever get cleared away and I won't know when to get rid of them so I please store the partial transactions on the user's computer and so they went away and came back with cookies so if you're wondering where cookies come from you can blame Mark no don't blame me so so that you know of course they go public in 1995 the stock goes through the roof and the dot boom is on the venture capitalist in san francisco were throwing money and anything that looks like it might have something to do with the internet and this goes on for a while 1998 in the midst of the dot boom Google gets started by Sergey Brin and Larry Page Yahoo! gets started a little bit before that to surf the internet the interesting thing about the arrival of a worldwide web is that it triggered an avalanche of content that flowed into the net it was so interesting people were not looking for money for the content they just wanted to know it was useful for somebody else kind of like what you do so this avalanche pours in and pretty soon nobody can find anything because there's so much of it so they need they need a search engine some of you will remember altavista which came out of the digital research labs on the west coast and then Yahoo! comes along which was kind of more manual I think than some of the others were and then Google of course with its clever strategy called page rank which was very successful it didn't have a business model to start with by the way there was no business model but not very long after it got started and after they brought Eric Schmidt in as the CEO sort of adult supervision then this three-way business model evolved which was quite successful I can get started in 1998 and the original idea was that John Postel would be the chief technology officer for ICANN and it would manage the domain name system and IP address allocations through the regional internet registries as I say John passed away in September of 1998 but ICANN was really needed and so they proceeded now one thing I have not done is to tell you about the 1978 to 1993 protocol wars between the open systems interconnection model and TCP IP to say nothing of X25 and X75 and X29 and so on that would take too long so I won't do that but a lot of people imagine that this is just smooth sailing and it was just step by step and everything was all planned out and they get nope it was Sturmundrang for many many years and it's still Sturmundrang today so just pushing a little further into time here a few other things the dot bust happens in 2000s around April big lesson there a lot of the startup CEOs apparently didn't understand the difference between revenue and capital and in economics 101 says you have a finite amount of capital to get your revenue engine going and if you don't get the revenue engine going you will run out of capital and then what so lots of dead bodies you know of startups in 2000 but the internet kept going the demand for that capability was still very strong so YouTube get started in 2005 Amazon web services comes up in 2006 the iPhone shows up I want to emphasize the iPhone for a moment because all of you I'm sure know what a transforming event that was but now this is anecdote number 17 or 18 I guess some of you may know that the mobile phone was invented by a guy named Marty Cooper he was working for Motorola at the time what you might not know is that he started working on it in 1973 which is when Bob and I start working on the internet and Metcalf starts working on something going on in 73 I don't know what it was whatever we were drinking so so anyway this thing gets started in the same year 73 and it gets turned on in 1983 the same year that the internet gets turned on so Danny Cohen another name you might know is very much influential in the internet's splitting of TCP and IP because of real-time operation he was all involved in packet speech among other things so Danny calls me up in early 1983 and says come have lunch I have something to show you so I show up and he's got this thing sitting on the table it's about this tall it's got a whip antenna on it and it weighs about two and a half pounds and what's that he says it's a phone and he said well where are the wires there aren't any how does it work so he says we talked about it for a while and he says I don't know the answer to that why don't you call the guy that invented it so I called Marty Cooper on a Motorola brick which is what we called it then and the first question I asked Marty was how long does the battery last and he says about 20 minutes but you can't hold the phone up longer than that anyway so so Marty blesses heart you know and presses on he's still around he's still around he's in his 90s now he's just written a book about the whole story of the invention of the mobile phone but the iPhone is really triggering as every one of you knows when jobs figured out something that none of us realized that we wanted which is a device that had a camera it had the access to the internet it had access to the telephone system it had a touch sensitive display I mean it's got all of these amazing features all of which existed as a technology never put together in such an interesting way that transforms everything because suddenly the internet is more accessible anywhere you can get a mobile signal you get to the internet and of course the mobile phone gets more useful because it gets access to all the applications that are running on the internet so the two are mutually reinforcing it's a really powerful event so 2007 is a big deal in the world that we now inhabit in 2008 2009 several developments came out of the academic world ethane open flow and NOx this is basically software defined networking which really has transformed the face of building networks today and in fact in 2010 Nick McEwen and his colleagues started a company called NICERA to build software defined networks and they were acquired by another company it's very successful there's lots more I'm not going to try to repeat the last 10 years of development and you've lived those last 10 years anyway so you know them as well as I do but they are pretty astonishing and you guys are a part of that so somebody asked me to look back and say what would you do differently? and so I decided I'd put a little list together the first one is obvious right I would have done IPv6 first instead of IPv4 but because it's been damn hard to cause an incompatible introduction of a new protocol at that low level in the architecture that in our own defense though we actually did a calculation when we were doing the original design of TCP to see how much address space we ought to have and remember it was an experiment and we didn't know if it was going to work so we said okay it has to work everywhere in the world because it's going to be supporting the defense department command and control system so we said okay so how many countries are there well how many how many networks per country and we thought well how about two so there would be some competition then we said how many countries are there and there wasn't any Google at the time to ask so we guessed it 128 because that's in power of two and you know we did the math 256 that's eight bits okay so we know how many networks we got a deal of 256 networks and then how many computers per network how about 16 million you know which is at the time they were millions of dollars and they didn't move anywhere they were in air conditioned rooms and they were hooked together with wires but we thought what the heck and besides it rounds it out to 32 bits which is cool so and we thought you know that's 4.3 billion terminations if you could if you could allocate them densely of course you never would but even if you could and that was more than there were people in the world at the time so we thought that ought to be enough for an experiment now I want you to imagine you know that you're a young vint serf in 1973 and you go and your future self goes back and whispers in your ear and says 128 bits of address space and your younger self says WTF that's that's 3.4 times 10 to the 38th addresses and you say yeah and I don't think I can sell that it doesn't pass the red face test your network has never even been demonstrated and you're telling me you need 10 to the 38th addresses so I don't know if I would have gotten away with that now there is a huge mistake that I made in for mobility support and this this one it's amazing how you can fool yourself into thinking you solved the problem when you actually haven't I remember splitting TCP and IP and then we had to figure out how the TCP identifiers would work on an end-to-end basis so we created a pseudo header sucking the IP addresses up out of the IP layer into the TCP layer and use that for socket identification and since I had an operating mobile radio network at the time I thought we had dealt with mobility except for one thing I didn't think about the possibility that your mobile would move from one network to a different network that had a different IP address space and at the time I was thinking okay I know I can put another address space at the TCP layer so that it's okay to switch IP out from under the TCP and those of you who know about Qwik know that the Qwik protocols establish a cryptographic shared variable at the Qwik layer so if an IP address changes not both but if one of them changes you can reconstruct the network if they both change of course they can't figure out who to talk to so that doesn't work anyway I thought that the mobile problem had been solved clearly it has not been solved and so that was a big mistake and I regret that but at the time I was patting myself on the back for saving bits in the header so you know be careful what you congratulate yourself for the other thing that has occurred to me is that radio has wonderful features one of which is broadcast you can transmit in all directions if you want to and we don't use that in any of our protocols in a serious way we kind of could do it especially with synchronous satellites you can imagine a protocol where a bunch of stuff gets sent to lots and lots of receivers and the guys that didn't get it could raise their hand and say please send me another copy but we never really built any protocols to take advantage of that maybe that's something we should think about another thing that we could have done is put crypto into the system sooner than we did and a lot of people come and say you know blankety blanky idiot why didn't you put more crypto in at the beginning and we wouldn't be in such a mess that we are today I don't actually think that's true but at the time in 1976 when Whit Diffie and Marty Hellman published their first paper on new directions in cryptography it was stunning and I'm sure the guys in the UK were especially stunned because they'd invented this idea in 1974 but they didn't tell anybody because they didn't want anybody to know about how clever this was so anyway the paper gets published and the next year or so in 78 or so the RSA algorithm to implement this idea gets invented now you could say why the hell didn't you just immediately implement the public key crypto and my reaction remember it's 1977 I'm trying to get the damn thing demonstrated and implemented on as many operating systems as possible and I look at the RSA idea and I said this is retrofitable I can't put this in anytime and so I wanted to get the system up and running and demonstrated first so we did that we were actually working using DES which is a conventional symmetric key to build a cryptographically secure system and we were demonstrating that with a program called Black Crypto Red I'm sure some of you know the red side of the net is a sensitive side the black side is post crypto and we stuck DES in the middle but key distribution is not nearly as nice with the symmetric keying system as it is with the public key system but anyway so we were clearly working on on the crypto support and of course the NSA was busily doing some of its development work as well but I remember thinking at the time okay if I were serious about insisting on cryptographic implementations who are the users of this thing it's graduate students and I don't mean any offense because I was a graduate student once too but I can't imagine graduate students being really good about key management and all the other things that you have to do and so it kind of was comfortable not doing that and not insisting on it too early in the game and of course multi-factor authentication would also have been a good thing to have thought about if we had any technology to support it because even then everybody knew the passwords were a terrible idea so here we are we really want to add more security into the system and so from my point of view we really need work on BGP I'm sure all of you know how easy it is to either be hijacked or just make a dumb mistake configuring something wrong almost all the really bad stuff that happens most of the time is somebody just making a mistake sometimes it's not somebody did it on purpose RPKI is another thing that I would like to see more implementation Ditto DNS seconding so those are all things that we should be working on I also think that more strong authentication in the system all the way down to identifying a hunk of hardware and being able to authenticate that it is what you think it is would be very helpful and the internet of things really really needs something to keep your simple IoT devices from being hijacked like the what is that half a million webcams that were hijacked to do the DDoS attack against Dyn which had cascade effects because Dyn was doing everybody's domain name resolution for a lot of very important companies and they all disappeared off the net because Dyn fell over which by the way hang on to that thought for just a second because if you're like me you may be needing a change of underwear because if you think about how dependent we are on certain key parts of the architecture and you can see when they don't work there are cascade failures if your mobile doesn't work batteries dead can't get a signal some other problem then you can't do two factor authentication maybe or you just can't get into your email so the business deal you were about to close doesn't happen more than once even just over this weekend I had to go through a whole series of login and authentication steps some of which involve the mobile I now get two devices that I have to make sure work everybody in this room knows that the probability of success is multiplicative so 90% successful times 90% successful is 81% successful if you rely on both things working and if there's a third thing it just gets worse so we are steering right now into a very fragile future in my opinion and I hope those of you who are thinking about software development architectures and things like that will really give some thought to making this more robust this isn't just a question of more security this is more resilience more alternatives like every screen ought to be useful for this authentications if you need it things like that now some people have criticized the basic assumption that we made that every device on the internet should be able to talk to every other device and the reason that we chose that as our principle was that we didn't know which devices we're going to need to talk to which devices at the time and so we had no rationale for inhibiting communications now today we look back and realize that since everything can talk to everything then the bad guys can talk to everything and they do and they cause trouble so it could very well be that we should reconsider how to hide parties from exposure and there are some suggestions and you know the sort of clean sheet designs that are coming to that I've already mentioned about the dependencies so and I'm also very conscious of the fact that I can't even read my own watch now those of you who have good eyes will know that this is a Ronald McDonald watch and I only have 10 minutes left I got this for teaching a class in networking at Hamburg University just outside of Illinois I am not kidding there is a university they teach people how to run McDonald's and they were about to network them all together so they could keep track of the sales and the use of resources and everything else instead they gave me a Ronald McDonald watch to commemorate the thing so anyway let me just riff for a second on open source and open standards because that's what you do and I tell you what you do is really important I still consider it to be one of the central engines of the evolution of the internet your willingness to share your code and your thoughts and your ideas and everything else it's wonderful think about what happened with the worldwide web there was no class in being a webmaster but what you got to do with the browser was say show source so you could see how did they make this really cool web page because you could see the html and so lots and lots of people learn how to be webmasters from each other and that's what you do you learn from each other which is great it accelerates the pace of development it also open source also gives you an opportunity to find bugs now there is a little problem with this sometimes open source leads people to think it's open source everybody's already found the bugs because it's open source therefore you don't need to look yes you can stop laughing now so I do worry about that and I have pushed very hard wherever I can to argue we need to support people like you more so that you can help us make more resilient and more stable and safe software it's really tough because the curation of this software is really important some of the bugs that some of you know lie around for 20 years and they just don't get noticed until they surface it exactly the wrong time there's lots of supply chain risk factors that are involved because the open source software could be anywhere in the stack and a bug that gets in deliberately or a malware that gets in deliberately can be really troublesome so there's a real challenge for us as a community to support open source software in a sustainable way so I just want you to know I'm a big fan of trying to find ways to do that we need this stuff the standards and the open source for interoperability some people say standards inhibit competition but I don't agree with that I think that having commonality and interoperability allows for competition on top of the standardized platform and of course we all need to remember that we want to adapt our software to whatever new platforms that come along Kubernetes and containers being a very good example virtual machines and so on this is a slide which I don't have time to talk about but I want to just pause and ask you to look at it for a second because these are all problem spots and they are not the full list they're just some of the challenges that lie ahead some of them are really tricky because they involve international agreements of some kind whether it's treaties or norms or something else think for a minute about the world that we wish we lived in one where accountability is enforceable and that agency is given to people and companies who are using the internet accountability and agency and in order to make that work you may have to give up some anonymity because if you can't hold people you can't hold anonymous people accountable and so if they're deliberately doing harmful things you have to have a way of tracking them down it's a little bit like license plates it's not a perfect analogy but you know license plates or govilliga except mine it says serves up so most of them most of them are just random stuff but the police department is permitted to penetrate the veil and find out who owns the car that may not be the person who was driving the car but they can penetrate that veil because it's their job to hold people accountable so I think you should think a little bit about that the other one the thing that the last two bullets especially I want to say straight to you that getting rid of bugs is really important making mistakes is easy in the software world we do dumb things you know like buffer overflows or off-by-one bugs or hey we just we just read a variable and did a compare and a branch on it except nobody ever set the variable so it's a random number that gives you really predictable behavior so there's an ethical component here every one of us who writes software has an ethical responsibility to do the best we can to make it safe and secure and reliable you know in all fairness the programming environments that we have are not exactly helpful in that endeavor I was going to say suck but you know but this is a challenge to the academics especially to figure out how to design and build programming environments that actually alert us to the dumb mistakes that we might make so I think this is the last slide final thoughts and I am happy to do some Q&A if you have any so first of all we have huge challenges to keep the internet open safe secure sustainable reliable and connected and the reason that this is a big challenge is that governments around the world recognizing that there are problems in the online world there are harms are being committed against corporations and people and they want to do something about it now some governments are trying to protect the government other governments are trying to protect the citizens but it's a big challenge when the governments are trying to enact laws that aren't necessarily implementable from the technical point of view so we're faced with trying to preserve the value of this connected internet while we're protecting people from harm I also think that your work an open source is important for digital preservation I want you to imagine you have a lot of digital stuff and you want it to last so that your great-grandchildren can have access to it well you know when you have digitized anything you need software to help interpret it most of the time what if it's a spreadsheet or a photograph if you don't if you can't run the code that created the digital object a hundred years from now then it may not be available so we have a huge challenge in maintaining the accessibility of digital objects over long periods of time there's something ironic about this there are clay tablets in cuneiform that were written four thousand or more years ago and if you can read cuneiform which maybe three or four people in the world can you can still read that clay tablet because it was a warehouse receipt and the warehouse burned down and the tablet was baked and that gave it longevity then there's vellum sheepskin calfskin and that stuff lasts easy a thousand years and if you keep going forward to recording media and then you get to five and a quarter inch floppies three and a half inch floppies CD rooms and they last for one two three four five six maybe 10 maybe 15 years how about seven track tape nine track tape so we have a big problem and that's preserving our digital future and that is going to mean that old software needs to keep running somehow interpreters emulators all those sorts of things uh so we really need to do that and the second of the last bullet talks about accessibility and by this I mean making software accessible to people with disabilities and I'm I wear hearing aids and I've worn them for about 65 70 years but there are people who have all kinds of you know vision problems and motor problems and everything else and they often get the short end of the stick when it comes to accessibility it's hard to do that it's not falling off a log you really have to think about how am I going to make this application work for somebody you can't see or can't hear or doesn't have fine grain motor control or some other problem but figuring out how to do that is worthwhile because we're losing out on talents of people just because you can't see doesn't mean your brain doesn't work and finally making open source sustainable and trustworthy is the task beholden to you so I apologize that I didn't leave any time unless you want to stick around for a bit for Q&A but I really thank you for listening and I thank you for what you're doing keep that thank you thank you I am confident that a few people will take your offer up for a Q&A okay as long as you're as it was as it was a genuine offer and you're ready to spend a few more minutes with us and it seems like it was so I think Hannah you are somewhere in the room with a microphone if you would like to ask a question of Dr. Surf please yeah let me let me warn you that if I have trouble hearing the question don't shout that doesn't help it's just a matter of clarity but the nice lady with the microphone is wondering any ills yep can I so we have the first usually I run around like her all though with the microphone but but if everybody's wearing a mask it doesn't do any good because you can't lip read through the mask so not so much a technical question but having done all of this and let your life story what's next oh well I didn't tell you about the project that started in 1998 at the jet propulsion laboratory just after the Pathfinder landed in 1997 after 20 years of failure to get to the Mars remember the two Vikings in 76 and then nothing works 97 this little rover lands successfully everybody cheers so I show up at JPL the next spring and I get together with a team that did the comms for for the Pathfinder and we spend about you know a couple of days together and we're trying to figure out what should we be doing now that is to say 1998 that's going to be needed 25 years from now and we said okay we want an interplanetary backbone network so we started working on the design of the solar system internet we are now at the point where we have new protocols that have been standardized by IETF and the consultative committee on space data systems we are running on the International Space Station have been there for a decade we've done deep space tests with some of the NASA spacecraft that have visited the comments and things like that we are prepared to go to the moon with the Artemis mission there's a Lunanet design that Goddard Space Flight Center is working we are working together with ESA and JAXA and Cary as well and of course NASA so this is all coming together there is a group if you want to look closely it's IPN Interplanetary Network IPN SIG Special Interest Group IPN SIG.org is a chapter of the internet society so we are located nowhere on earth we are in the rest of the solar system I've been trying to trying to get JPL they issue me a badge that says Resident Alien which would be really cool fun fact several of the folks that running our AV here today are involved in some of the fun and exciting missions at NASA JPL so thank you all for running those cameras that are letting us share this talk on the internet so we've got one more question in the back yeah so there was a point on the previous slide talking about accountability and traceability accountability and traceability or something of that sort right and it's talking about how we wanted to to make sure that people were behaving well and to protect you know corporations governments and individuals from these bad actors a better question from my perspective is how do we protect ourselves from government bad actors also very good question again interestingly enough accountability makes sense in that context too because we would expect governments to be accountable for what they've done not all governments are willing to be accountable you and I would probably agree on that but if we establish a practice of accountability and we insist that that's part of the architecture and design then at least we have a shot at it I would not stand here and say I promise or guarantee that that will work but I believe that we should hold ourselves to the philosophy that people should be accountable and government should be accountable for what they do and of course one way to do that is to vote Hi so it's starting to look like with the rise of VR technology and AR that we're going to be in a lot more of virtual worlds but we're having a lot of problems just deciding on what standards to use and even the underlying communications technologies do you have any advice tips tricks that we could possibly use while creating these? So remember I probably don't have any better idea than you do so be careful whatever I say you should take with a grain of salt I had the impression that there were some really interesting ideas a while back remember Second Life and VRML now I don't know VRML intimately so I could be saying look at that thing and you would say I looked at it and it's a pile of crap but there have been attempts to find ways of sharing the descriptions of multi-dimensional spaces I think we're going to have to experiment with this a lot here's the one thing I'm hoping for lots of people are going to try 3D stuff because it's all very exciting we got chipsets that can do it we got headsets oh that reminds me this is going to be weird you know we've been spending the last two years doing the video conferencing and Google Meet and all these other things and people are saying yeah I can hardly wait to do this with the 3D you know headsets and I'm sitting here thinking how's that going to look because you got the camera here but you're wearing this thing and so you look like Darth Vader in fact we all look like Darth Vader and so in order to do this 3D conferencing you're going to have to have an avatar which means it's a new business model right you can have my avatar can have hair for example and you know you can rent suits from you know Ralph Lauren I think we're a long ways from getting this right the one thing I will notice is that the headsets have got to the point where there is less problem you know people get nauseated because they get signals proprioceptive signaling isn't that a great 50 cent word the proprioceptive signaling doesn't match your eyes are telling you're going like this in your bodies as you're standing here like that and your brain says this doesn't make any sense and so so then it makes you throw up which is you know thanks a lot so so I think we still have ways to go but it's getting better but I I think there's a whole ecosystem waiting and a lot of design and standardization I would love it if it turns out that those three dimensional environments could inter work somehow but that would require a lot of agreement and cooperation and commonality of you know headset capability and coding and everything else so guess what you get to try that out this is going to be cool we got one one more here Hannah it's up in front and then and then one on the side well you can yell but we want other people to hear you too so so you mentioned supply chain issues and specifically provenance software provenance do you have any thoughts on systems like Nix Gwix or Gytian have you heard of any of those those are not familiar so if you want to say a little more about what they are supposed to do is that well Nix is about reproducible build systems all the way to the executable okay so essentially your hash is part of the binary okay yes yes got it so I like that a lot I mean I like the fact that it's that you can't make alterations without they're being visible you can make the alterations but if you check first there are a lot of hardware boot systems do that right so I'm a big fan of that sort of thing it's another example of trying to make sure that the devices are equipped with the ability to figure out where did this stuff come from has it maintained integrity on his journey before I actually loaded in and booted up so I like that idea a lot I think we have one other question over there and then we need to wrap up no problem so thank you thank you for answering my question this is actually confirming an anecdote that Professor Douglas Comer told our class on the construction of the TCV IP protocol when in discussions with the DoD and that one of them thing was running around a city that has been destroyed by nuclear war and I was wondering if you could confirm or deny any of that right okay so there's great confusion in the history here Paul Barron when he was at RAND Corporation 1962 to 64 published an 11 volume series called on Distributed Communication he was talking about mesh networks this is 62 mesh networks and packetized didn't call it that call it message blocks for digital speech he imagined routers he didn't call them routers either relays on the tops of telephone poles across the country and so this big mesh could you could blow holes in it as long as there was some connectivity stuff would get through used hot potato routing which basically was send it to everybody and you know try to get it try to get it you know to the destination so that was his model and never got built but it was published then Arpanet comes along now the Arpanet was driven by an economic requirement Arpanet was spending money on a dozen universities to do artificial intelligence and computer science research in the 1960s and every year everybody said you have to buy us a new brand new world-class computers so we can keep doing world-class research and Arpanet said we can't afford that so they said we're going to build a network and you can share and everybody hated that but they said we're going don't don't worry we are going to fund all of you so you don't have to hide your results in order to have an edge on next year's proposals we want you to share your results share your software and share your computing capability so we can accelerate the pace of artificial intelligence and computer science so they in fact did build the network but it was not based on nuclear holocaust or anything it was just based on trying out a packet switching technology which we believed at the time would work a lot better than dial up a computer send some data hang up dial up another computer send some data and hang up we didn't think circuit switching was going to work for our bursty kind of applications of course the telecom cannon of the day was of course you do circuit switching that's how we've been doing it for 100 whatever it was 70 years or 50 years and so we asked AT&T would you like to participate in this thing they said no it won't work but we'll sell you dedicated circuits if you like so you can build your stupid network and so they did and we did and it worked so now when the internet comes along I was worried about exactly the problem of recovery from major failures but and we even did an experiment to figure out what happens if you had a partition network Radia Perlman figured out how to do the routing system to recover from a perforated or bifurcated network but all in all honesty none of those protocols were ever tested in the kind of nuclear holocaust that really blew a lot of pieces apart I actually did however just to demonstrate it fly packet radios in the strategic air command bombers and and and basically cut up pieces of the Arpanet and then glued them back together using ground and air based mobile packet radios just to demonstrate the TCP IP would link the pieces back together again but it wouldn't it would was not mature enough to deal with a serious post nuclear scenario so the real answer is none of this was really built to do that it was built to figure out how to get computers to talk to each other okay got to wrap it up but thanks again really appreciate you have a good evening thank you for joining us and for for sharing with us and and for helping to you know be part of creating many of our careers so so with that scale is officially over at least for those of you that are attendees for those of us that are wearing orange shirts and jerseys and and other embroidered materials we have a couple more hours of four days of cleanup and and wrap up if you want to make that easier for us next year we are always looking for additional volunteers and team members but with that we will see you again March 9th through the 12th and Pasadena it's only about six months away which I think just I think I just scared my team quite a bit nine months away they're like they didn't want me to steal three months you can be part of that make this make this journey easier for the rest of us thank you very much